Most efficient way to append arrays in C#?

I am pulling data out of an old-school ActiveX in the form of arrays of doubles. I don't initially know the final number of samples I will actually retrieve.

What is the most efficient way to concatenate these arrays together in C# as I pull them out of the system?

You can't append to an actual array - the size of an array is fixed at creation time. Instead, use a List<T> which can grow as it needs to.

Alternatively, keep a list of arrays, and concatenate them all only when you've grabbed everything.

See Eric Lippert's blog post on arrays for more detail and insight than I could realistically provide :)

Concatenating arrays is simple using linq extensions which come standard with .Net 4

Biggest thing to remember is that linq works with IEnumerable<T> objects, so in order to get an array back as your result then you must use the .ToArray() method at the end

Example of concatenating two byte arrays:

byte[] firstArray = {2,45,79,33};
byte[] secondArray = {55,4,7,81};
byte[] result = firstArray.Concat(secondArray).ToArray();

I believe if you have 2 arrays of the same type that you want to combine into a third array, there's a very simple way to do that.

here's the code:

String[] theHTMLFiles = Directory.GetFiles(basePath, "*.html");
String[] thexmlFiles = Directory.GetFiles(basePath, "*.xml");
List<String> finalList = new List<String>(theHTMLFiles.Concat<string>(thexmlFiles));
String[] finalArray = finalList.ToArray();

I recommend the answer found here: https://stackoverflow.com/questions/1547252/how-do-i-concatenate-two-arrays-in-c

e.g.

var z = new int[x.Length + y.Length];
x.CopyTo(z, 0);
y.CopyTo(z, x.Length);

The solution looks like great fun, but it is possible to concatenate arrays in just two statements. When you're handling large byte arrays, I suppose it is inefficient to use a Linked List to contain each byte.

Here is a code sample for reading bytes from a stream and extending a byte array on the fly:

byte[] buf = new byte[8192];
byte[] result = new byte[0];
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
Array.Resize(ref result, result.Length + count);
Array.Copy(buf, 0, result, result.Length - count, count);
}
}
while (count > 0); // any more data to read?
resStream.Close();

> using this we can add two array with out any loop.

I believe if you have 2 arrays of the same type that you want to combine into one of array, there's a very simple way to do that.

Here's the code:

String[] TextFils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
String[] finalArray = TextFils.Concat(ExcelFils).ToArray();

or

String[] Fils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
Fils = Fils.Concat(ExcelFils).ToArray();

If you can make an approximation of the number of items that will be there at the end, use the overload of the List constuctor that takes count as a parameter. You will save some expensive List duplications. Otherwise you have to pay for it.

You might not need to concatenate end result into contiguous array. Instead, keep appending to the list as suggested by Jon. In the end you'll have a jagged array (well, almost rectangular in fact). When you need to access an element by index, use following indexing scheme:

double x = list[i / sampleSize][i % sampleSize];

Iteration over jagged array is also straightforward:

for (int iRow = 0; iRow < list.Length; ++iRow) {
  double[] row = list[iRow];
  for (int iCol = 0; iCol < row.Length; ++iCol) {
    double x = row[iCol];
  }
}

This saves you memory allocation and copying at expense of slightly slower element access. Whether this will be a net performance gain depends on size of your data, data access patterns and memory constraints.

Here is a usable class based on what Constantin said:

class Program
{
    static void Main(string[] args)
    {
        FastConcat<int> i = new FastConcat<int>();
        i.Add(new int[] { 0, 1, 2, 3, 4 });
        Console.WriteLine(i[0]);
        i.Add(new int[] { 5, 6, 7, 8, 9 });
        Console.WriteLine(i[4]);

        Console.WriteLine("Enumerator:");
        foreach (int val in i)
            Console.WriteLine(val);

        Console.ReadLine();
    }
}

class FastConcat<T> : IEnumerable<T>
{
    LinkedList<T[]> _items = new LinkedList<T[]>();
    int _count;

    public int Count
    {
        get
        {
            return _count;
        }
    }

    public void Add(T[] items)
    {
        if (items == null)
            return;
        if (items.Length == 0)
            return;

        _items.AddLast(items);
        _count += items.Length;
    }

    private T[] GetItemIndex(int realIndex, out int offset)
    {
        offset = 0; // Offset that needs to be applied to realIndex.
        int currentStart = 0; // Current index start.

        foreach (T[] items in _items)
        {
            currentStart += items.Length;
            if (currentStart > realIndex)
                return items;
            offset = currentStart;
        }
        return null;
    }

    public T this[int index]
    {
        get
        {
            int offset;
            T[] i = GetItemIndex(index, out offset);
            return i[index - offset];
        }
        set
        {
            int offset;
            T[] i = GetItemIndex(index, out offset);
            i[index - offset] = value;
        }
    }

    #region IEnumerable<T> Members

    public IEnumerator<T> GetEnumerator()
    {
        foreach (T[] items in _items)
            foreach (T item in items)
                yield return item;
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.

For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.

I had the same issue to solve with the requirement of appending a specific count instead of the whole array, and my first solution was the same as suggested by Hugo. But my feeling said "inefficient" because of that many resizes.

Then I remembered that the StringBuilder is capacity-optimized. As next I asked myself, does it apply to MemoryStream too. After some tries I can say yes it does.

The MemoryStream starts with a minimal capacity of 256 bytes and grows if necessary by the double of its last capacity, like 256, 512, 1024, 2048, 4096, 8192 and so on.

My next question was, how long it takes to do array resize and copy in contrast to using a MemoryStream. Using a MemoryStream was much faster instead of array resize and copy.

Hence, I guess using a MemoryStream is the most efficient way.