# Parallel Programming

Course: [C# Advanced](https://www.devtutorials.ai/learn/advanced-csharp) · Section: Multithreading
Source: https://www.devtutorials.ai/learn/advanced-csharp/multithreading-parallel/parallel-programming

> Use every CPU core: Parallel.For, ForEach and Invoke, thread-local data, PLINQ, degree of parallelism, stopping loops, errors, performance and concurrency.

## Introduction to Parallel Programming

Think about marking 400 exam papers.

One teacher marking alone takes all weekend. Four teachers who split the pile, 100 papers each, finish in an afternoon. They each do the same kind of work, just on different papers, at the same time.

That is **parallel programming**: splitting one big job into pieces and running the pieces **at the same time** on different CPU cores.

**In simple words:** parallel programming uses all of your computer's cores to finish heavy calculations faster.

Modern computers have 4, 8, 16, or more cores. A normal `for` loop uses only one of them. The rest sit idle. The **`Parallel`** class and **PLINQ** let you use all of them with just a few lines of code.

**Easy Way to Remember:** a `for` loop is **one teacher**. `Parallel.For` is **a team of teachers** sharing the pile.

## What You Will Learn

1. The `Parallel` class: `Parallel.For`, `Parallel.ForEach`, and `Parallel.Invoke`.
2. Thread-local data, and how to combine parallel results.
3. PLINQ: parallel LINQ with `AsParallel()`.
4. How to control the degree of parallelism.
5. How to stop or break out of a parallel loop.
6. How exceptions work in parallel code.
7. When parallel code is faster, and when it is not.
8. The difference between parallel programming and concurrency.

**About the Run buttons in this lesson.** The browser playground has only **one** core, so parallel loops run their iterations one after another here. The results are exactly the same as on a real computer with multiple cores; only the speed differs. To see the real speed improvement, copy the examples into a console app on your own computer and run them there.

## Start With a Working Example

Press **Run**.

```csharp runnable title="Check 8 images, using every core"
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Threading.Tasks;

string[] images = { "a.jpg", "b.jpg", "c.jpg", "d.jpg", "e.jpg", "f.jpg", "g.jpg", "h.jpg" };
var sizes = new ConcurrentDictionary<string, int>();

Parallel.ForEach(images, image =>
{
    sizes[image] = FakeResize(image);   // heavy work on one image
});

foreach (var pair in sizes.OrderBy(p => p.Key))
    Console.WriteLine($"{pair.Key}: {pair.Value} KB");

static int FakeResize(string name)
{
    int work = 0;
    for (int i = 0; i < 200_000; i++) work += i % 7;
    return 100 + name[0] - 'a';
}
```

`Parallel.ForEach` distributes the images among several workers. On an 8-core computer, all eight can be processed at the same time. The results are stored in a `ConcurrentDictionary` because several workers write to it simultaneously.

### Try This

Add `"i.jpg"` to the array. A ninth line appears: `i.jpg: 108 KB`.

## Parallel Class

The **`Parallel`** class (in `System.Threading.Tasks`) runs work on many cores at once. It has three main methods:

| Method | Runs | Like |
|---|---|---|
| `Parallel.For(from, to, body)` | The body once for each number | A `for` loop |
| `Parallel.ForEach(items, body)` | The body once for each item | A `foreach` loop |
| `Parallel.Invoke(a, b, c)` | Several different methods at the same time | Calling `a(); b(); c();` |

Three important differences from normal loops:

- The iterations run in **no fixed order**. Different iterations may finish at different times.
- The call **blocks** until every iteration has finished. When it returns, all the work is done.
- The body must be **safe to run at the same time** as itself. Do not change shared variables unless you protect them.

## Parallel.For

**`Parallel.For(fromInclusive, toExclusive, i => { ... })`** works like `for (int i = from; i < to; i++)`, but spreads the iterations across multiple cores.

```csharp runnable title="Fill an array of results in parallel"
using System;
using System.Threading.Tasks;

double[] roots = new double[10];

Parallel.For(0, roots.Length, i =>
{
    roots[i] = Math.Sqrt(i);   // each i writes its own slot: safe
});

for (int i = 0; i < roots.Length; i++)
    Console.WriteLine($"sqrt({i}) = {roots[i]:0.000}");
```

Writing to `roots[i]` is safe because each iteration writes to a **different slot**. The `to` value is exclusive, just like `i < roots.Length` in a normal loop.

## Parallel.ForEach

**`Parallel.ForEach(items, item => { ... })`** works like `foreach`, but spreads the work across cores. It works with any collection: arrays, lists, dictionaries, or lines from a file.

```csharp runnable title="Score every student in parallel"
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

var students = new List<(string Name, int[] Marks)>
{
    ("Asha",  new[] { 78, 85, 92 }),
    ("Ravi",  new[] { 65, 70, 58 }),
    ("Meena", new[] { 90, 95, 88 }),
    ("Kiran", new[] { 40, 55, 61 }),
};

var report = new ConcurrentBag<string>();

Parallel.ForEach(students, student =>
{
    double average = student.Marks.Average();
    string grade = average >= 85 ? "A" : average >= 65 ? "B" : "C";
    report.Add($"{student.Name,-6} {average,5:0.0}  {grade}");
});

foreach (string line in report.OrderBy(l => l))
    Console.WriteLine(line);
```

Because iterations finish in any order, the results go into a `ConcurrentBag` and are sorted before printing.

## Parallel.Invoke

**`Parallel.Invoke`** runs several **different** pieces of work at the same time, and waits for all of them to finish.

```csharp runnable title="Build three parts of a report at once"
using System;
using System.Threading.Tasks;

string summary = "", chart = "", table = "";

Parallel.Invoke(
    () => summary = BuildSummary(),
    () => chart = BuildChart(),
    () => table = BuildTable());

Console.WriteLine(summary);
Console.WriteLine(chart);
Console.WriteLine(table);

static string BuildSummary() => "Summary: sales up 12%";
static string BuildChart() => "Chart: 12 bars drawn";
static string BuildTable() => "Table: 340 rows";
```

Each action writes its own variable, so there is no shared data to protect. When `Parallel.Invoke` returns, all three pieces of work are complete.

## Thread-Local Data

The body of a parallel loop runs on several threads at the same time. Treat every variable outside the loop as **shared**: only write to it with `Interlocked`, a `lock`, or a concurrent collection.

Imagine summing a million numbers with `Parallel.For`. If every iteration adds to one shared `total` variable using a `lock`, the workers spend most of their time waiting for the lock instead of doing work.

The better way is **thread-local data**: each worker keeps its **own** running total. At the end, each worker's total is combined into the shared result, using a lock just **once** per worker instead of a million times.

`Parallel.For` has a special overload for this, with three steps:

| Step | Runs | Job |
|---|---|---|
| `localInit` | Once per worker, at the start | Create this worker's own total variable |
| `body` | Once per iteration | Add to this worker's total, then return it |
| `localFinally` | Once per worker, at the end | Add this worker's total to the shared result, safely |

```csharp runnable title="Each worker keeps its own subtotal"
using System;
using System.Threading;
using System.Threading.Tasks;

long total = 0;

Parallel.For(
    1, 1_000_001,
    () => 0L,                                    // localInit: my own subtotal
    (i, state, subtotal) => subtotal + i,        // body: add to my subtotal
    subtotal => Interlocked.Add(ref total, subtotal)); // localFinally: combine once

Console.WriteLine($"Sum of 1..1,000,000 = {total}");
```

The shared `total` is touched only once per worker, not a million times, so there is almost no waiting for locks.

There is also a **`ThreadLocal<T>`** class, which gives every thread its own private copy of a value. This is useful for objects that are not thread-safe, such as `Random` in older .NET versions:

```text
var random = new ThreadLocal<Random>(() => new Random());
Parallel.For(0, 100, i => Console.WriteLine(random.Value!.Next(1, 7)));
```

## Aggregating Parallel Results

**Aggregating** means combining many separate results into one final result: a total, a count, a maximum, or a list.

| Way | Use it when |
|---|---|
| Write to `results[i]` in an array | One result per item, and you know in advance how many items there are |
| Concurrent collection (`ConcurrentBag`, `ConcurrentDictionary`) | An unknown number of results, or results grouped by key |
| Thread-local subtotal + `Interlocked` or `lock` | Computing a single value like a sum, count, minimum, or maximum |
| PLINQ `Sum()`, `Max()`, `Aggregate()` | The simplest way when a LINQ query already fits your needs |

```csharp runnable title="Count, max and grouped totals from one parallel loop"
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

int[] temperatures = { 31, 35, 29, 38, 33, 41, 36, 30, 28, 39 };

int hotDays = 0;
int maxTemp = int.MinValue;
object gate = new object();
var byBand = new ConcurrentDictionary<string, int>();

Parallel.ForEach(temperatures, t =>
{
    if (t >= 35) Interlocked.Increment(ref hotDays);
    lock (gate) { if (t > maxTemp) maxTemp = t; }
    string band = t >= 35 ? "hot" : t >= 30 ? "warm" : "mild";
    byBand.AddOrUpdate(band, 1, (_, n) => n + 1);
});

Console.WriteLine($"Hot days: {hotDays}");
Console.WriteLine($"Highest: {maxTemp}");
foreach (var b in byBand.OrderBy(b => b.Key))
    Console.WriteLine($"{b.Key}: {b.Value}");
```

## PLINQ

**PLINQ** (Parallel LINQ) runs a LINQ query on several cores at the same time. You add one method call, **`.AsParallel()`**, and the rest of the query stays the same.

```csharp runnable title="LINQ, then the same query with AsParallel"
using System;
using System.Linq;

int[] numbers = Enumerable.Range(1, 100_000).ToArray();

int normal = numbers.Where(IsPrime).Count();
int parallel = numbers.AsParallel().Where(IsPrime).Count();
Console.WriteLine($"Primes (LINQ):  {normal}");
Console.WriteLine($"Primes (PLINQ): {parallel}");

int[] firstFive = numbers.AsParallel()
                         .AsOrdered()          // keep the original order
                         .Where(IsPrime)
                         .Take(5)
                         .ToArray();
Console.WriteLine($"First five primes: {string.Join(", ", firstFive)}");

long sumOfSquares = numbers.AsParallel().Sum(n => (long)n * n);
Console.WriteLine($"Sum of squares: {sumOfSquares}");

static bool IsPrime(int n)
{
    if (n < 2) return false;
    for (int d = 2; d * d <= n; d++)
        if (n % d == 0) return false;
    return true;
}
```

| Method | What it does |
|---|---|
| `AsParallel()` | Turns the query that follows into a parallel query that runs on multiple cores |
| `AsOrdered()` | Keeps results in the original order (a little slower) |
| `AsSequential()` | Goes back to normal, one-core-at-a-time LINQ |
| `WithDegreeOfParallelism(n)` | Uses at most `n` cores |
| `ForAll(action)` | Runs an action on each result in parallel, without collecting them first |

By default, PLINQ results come back in **any order**. Add `AsOrdered()` if order is important, or sort the results at the end.

## Degree of Parallelism

The **degree of parallelism** is the number of iterations that may run at the same time. By default, .NET decides automatically, usually about one iteration per core.

You can set a limit with **`ParallelOptions.MaxDegreeOfParallelism`**, or `WithDegreeOfParallelism` in PLINQ. Reasons to limit it:

- Leave some cores free for other work, such as handling a web server's requests.
- Protect a shared resource, such as a disk or database that cannot handle many connections at once.
- Make testing and debugging easier.

```text
using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

int running = 0, maxRunning = 0;
object gate = new object();
var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };

Parallel.For(0, 20, options, i =>
{
    lock (gate) { running++; maxRunning = Math.Max(maxRunning, running); }
    int work = 0;
    for (int k = 0; k < 100_000; k++) work += k % 3;
    lock (gate) { running--; }
});

Console.WriteLine($"Never more than 2 at once: {maxRunning <= 2}");

int[] evens = Enumerable.Range(1, 10).AsParallel()
                        .WithDegreeOfParallelism(2)
                        .Where(n => n % 2 == 0)
                        .OrderBy(n => n)
                        .ToArray();
Console.WriteLine($"Evens: {string.Join(", ", evens)}");
```

This example needs real worker threads, so it is shown as plain code. In the single-threaded browser it would wait forever for a second worker that never comes. Run it in a console app on your own computer.

Setting the degree **higher** than the number of cores available does not make CPU work faster. The extra workers just take turns using the available cores.

## Stopping and Breaking Parallel Loops

A normal loop uses `break` to stop early. A parallel loop cannot use `break` because several iterations are running at the same time. Instead, the body can take a **`ParallelLoopState`** parameter and call:

| Method | Meaning |
|---|---|
| `state.Stop()` | Stop as soon as possible. Iterations already running finish; no new ones start |
| `state.Break()` | Finish every iteration **before** this index, but none after it |

The loop returns a result that tells you how it ended:

| Property | Meaning |
|---|---|
| `IsCompleted` | `true` if every iteration ran without stopping |
| `LowestBreakIteration` | The index where `Break()` was called, or `null` |

```csharp runnable title="Stop as soon as a match is found"
using System;
using System.Threading.Tasks;

int[] codes = { 12, 47, 33, 99, 15, 99, 64 };
bool found = false;

ParallelLoopResult result = Parallel.For(0, codes.Length, (i, state) =>
{
    if (codes[i] == 99)
    {
        found = true;
        state.Stop();     // no need to look any further
    }
});

Console.WriteLine($"Found 99: {found}");
Console.WriteLine($"Loop ran to the end: {result.IsCompleted}");
```

```csharp runnable title="Break: everything before the first big number"
using System;
using System.Threading.Tasks;

int[] readings = { 3, 5, 8, 120, 6, 4 };

ParallelLoopResult result = Parallel.For(0, readings.Length, (i, state) =>
{
    if (readings[i] > 100) state.Break();
});

Console.WriteLine($"First bad reading at index: {result.LowestBreakIteration}");
Console.WriteLine($"Every reading before it was processed: {result.LowestBreakIteration == 3}");
```

To stop a parallel loop from **outside** (for example, when the user clicks a Cancel button), pass a `CancellationToken` in `ParallelOptions`. The loop then throws `OperationCanceledException`:

```text
var options = new ParallelOptions { CancellationToken = cts.Token };
try
{
    Parallel.ForEach(files, options, file => Process(file));
}
catch (OperationCanceledException)
{
    Console.WriteLine("Cancelled by the user");
}
```

## Exception Handling in Parallel Operations

If an iteration throws an exception, the parallel loop stops starting new iterations, waits for the currently running ones to finish, and then throws an **`AggregateException`**. This exception holds **every** exception that happened during the loop, because several iterations may have failed at the same time.

```csharp runnable title="Two bad records, one AggregateException"
using System;
using System.Threading.Tasks;

string[] records = { "10", "20", "abc", "40", "xyz" };

try
{
    Parallel.ForEach(records, record =>
    {
        if (!int.TryParse(record, out _))
            throw new FormatException($"Bad record: {record}");
    });
}
catch (AggregateException ae)
{
    Console.WriteLine($"Failures: {ae.InnerExceptions.Count > 0}");
    foreach (Exception ex in ae.InnerExceptions)
        Console.WriteLine($"  {ex.GetType().Name}: {ex.Message.StartsWith("Bad record")}");
}
```

Because the loop stops early, the number of failures recorded can vary on a multi-core computer. Always loop over `InnerExceptions` instead of assuming there is exactly one exception.

If you want **every** item to be processed even when some fail, catch the exception inside the body and record the error instead of throwing:

```csharp runnable title="Keep going: collect errors instead of stopping"
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Threading.Tasks;

string[] records = { "10", "20", "abc", "40", "xyz" };
var good = new ConcurrentBag<int>();
var errors = new ConcurrentBag<string>();

Parallel.ForEach(records, record =>
{
    try
    {
        good.Add(int.Parse(record));
    }
    catch (FormatException)
    {
        errors.Add(record);
    }
});

Console.WriteLine($"Good total: {good.Sum()}");
Console.WriteLine($"Bad records: {string.Join(", ", errors.OrderBy(e => e))}");
```

PLINQ works the same way: an exception inside the query is thrown as an `AggregateException` when you read the results.

## Parallel Programming Performance

Parallel code is not automatically faster. Splitting the work, starting workers, and combining results all take time. Parallelism only helps when there is **enough work per item** to be worth the overhead.

| Situation | Parallel helps? | Why |
|---|---|---|
| Heavy calculation on many items (images, prime checks, simulations) | **Yes** | Each item takes long enough to justify splitting |
| Tiny work per item (adding two numbers) | **No** | The overhead of splitting takes more time than the work |
| Few items (under about 100 light items) | **No** | Not enough items to benefit from splitting |
| Waiting for disk, network, or database | **No** | Use `async`/`await`; more cores do not make a server respond faster |
| Every iteration locks the same object | **No** | Workers wait for the lock instead of working in parallel |
| Only one core available | **No** | There is no second core to run the parallel work on |

Rules for fast parallel code:

- Give each iteration a reasonable amount of work, not just a few instructions.
- Avoid shared state; use thread-local data and combine once at the end.
- Avoid locks inside the loop body where possible.
- **Measure** with `Stopwatch` on your real computer with real data. Never assume parallelism will help.

```text
var sw = Stopwatch.StartNew();
for (int i = 0; i < images.Length; i++) Resize(images[i]);
Console.WriteLine($"Normal:   {sw.ElapsedMilliseconds} ms");

sw.Restart();
Parallel.For(0, images.Length, i => Resize(images[i]));
Console.WriteLine($"Parallel: {sw.ElapsedMilliseconds} ms");
```

On an 8-core computer, heavy CPU work often runs 4 to 7 times faster in parallel. It is never quite 8 times faster because some time is always spent on coordination and overhead.

## Parallel Programming vs Concurrency

These two words are often confused, but they mean different things.

**Concurrency** means several tasks are **in progress** at the same time. They may take turns on one core. It is about **structure**: managing multiple tasks that may start, run, and finish in an interleaved way.

**Parallelism** means several tasks are **running** at the exact same instant, on different cores. It is about **speed**: doing multiple things at the same physical moment to get work done faster.

Think of a chef. A single chef who chops vegetables while the soup simmers and the bread bakes is **concurrent**: multiple dishes in progress with one pair of hands, working on them one at a time. Four chefs each chopping a different vegetable at the exact same moment are **parallel**: the same work happening on different people at the same time.

| | Concurrency | Parallelism |
|---|---|---|
| Meaning | Many tasks in progress, maybe taking turns | Many tasks running at the exact same instant |
| Needs multiple cores | No | Yes |
| Main goal | Stay responsive while waiting | Finish heavy calculations faster |
| Typical work | I/O-bound: web requests, files, databases | CPU-bound: images, math, data processing |
| C# tools | `async`/`await`, `Task.WhenAll`, channels | `Parallel`, PLINQ, `Task.Run` |

This browser playground demonstrates the difference: with only one core, it can run **concurrent** code (Chapter 1's async examples interleave their waits), but it cannot run anything truly **in parallel**.

**Easy Way to Remember:** concurrency is **juggling** (one person, many balls); parallelism is **more hands** (multiple people working at once).

Use concurrency for **waiting** work and parallelism for **calculating** work. A web server mostly needs concurrency. A video encoder mostly needs parallelism. Many real-world applications need both.

## Common mistakes

| Mistake | Wrong | Correct | Why |
|---|---|---|---|
| Shared total without protection | `total += i;` in the body | Thread-local subtotal + `Interlocked.Add` | Updates get lost when multiple threads update at once |
| Plain `List` in the body | `list.Add(x);` from many threads | `bag.Add(x);` (ConcurrentBag) | `List<T>` is not thread-safe |
| Expecting order | Printing inside `Parallel.ForEach` | Collect results, then sort or use `AsOrdered()` | Iterations finish in any order |
| Parallel for tiny work | `Parallel.For(0, 10, i => sum[i] = i + 1)` | Use a normal `for` loop | Overhead of parallelism is bigger than the work |
| Parallel for I/O | `Parallel.ForEach(urls, u => Download(u))` | `await Task.WhenAll(urls.Select(DownloadAsync))` | Threads just sit idle waiting for the network |
| Catching the wrong exception | `catch (FormatException)` around the loop | `catch (AggregateException ae)` | Parallel loops wrap all errors in an AggregateException |
| Lock on every iteration | `lock (g) { total += i; }` on each iteration | Use thread-local subtotals, combine once per worker | Workers queue up behind the lock instead of running in parallel |
| Blocking the UI thread | `Parallel.For(...)` in a button click handler | `await Task.Run(() => Parallel.For(...))` | `Parallel` methods block until all work finishes, freezing the UI |

## What to Remember

- Parallel programming splits heavy CPU work across all cores to finish it faster.
- `Parallel.For` and `Parallel.ForEach` are parallel loops; `Parallel.Invoke` runs several different tasks at once. They all wait until the work is done.
- Iterations run in no fixed order, and the body must be safe to run many times at the same time.
- Thread-local data (`localInit`, `localFinally`) lets each worker keep its own subtotal and combine once.
- Collect results in array slots, concurrent collections, thread-local totals, or use PLINQ.
- PLINQ is LINQ plus `.AsParallel()`; add `AsOrdered()` if order matters.
- `MaxDegreeOfParallelism` limits how many iterations run at the same time.
- `state.Stop()` ends a loop as soon as possible; `state.Break()` finishes every iteration before the break point.
- Parallel loops throw an `AggregateException` containing every error that occurred.
- Parallel only pays off when there is enough CPU work per item; always measure it.
- Concurrency is many tasks in progress (juggling); parallelism is many tasks at the same instant (more hands).

Next: Advanced Generics