Byte array of unknown length in java
JavaBufferByteJava Problem Overview
I am constructing an array of bytes in java and I don't know how long the array will be.
I want some tool like Java's StringBuffer that you can just call .append(byte b) or .append(byte[] buf) and have it buffer all my bytes and return to me a byte[] when I'm done. Is there a class that does for bytes what StringBuffer does for Strings? It does not look like the ByteBuffer class is what I'm looking for.
Anyone have a good solution?
Java Solutions
Solution 1 - Java
Try ByteArrayOutputStream. You can use write( byte[] ) and it will grow as needed.
Solution 2 - Java
Just to extend the previous answer, you can use ByteArrayOutputStream and it's method public void write(byte[] b, int off, int len), where parameters are:
> b - the data > > off - the start offset in the data > > len - the number of bytes to write
If you want to use it as a "byte builder" and insert byte by byte, you can use this:
byte byteToInsert = 100;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.write(new byte[]{byteToInsert}, 0, 1);
Then you can use baos.toString() method to convert the array to string. The advantage is when you need to set up encoding of input, you can simply use i.e.:
baos.toString("Windows-1250")
Solution 3 - Java
I wrote one that is really easy to use and avoids a lot of byte array buffer copying.
It has one method called add.
You can add strings, bytes, byte, long, int, double, float, short, and chars to it.
The API is easy to use and somewhat fail safe. It does not allow you to copy the buffer around and does not promote having two readers.
It has a bounds check mode and a I KNOW WHAT I AM DOING MODE with no bounds checking.
The bounds check mode auto-grows it so there is no hassle.
https://github.com/RichardHightower/boon/wiki/Auto-Growable-Byte-Buffer-like-a-ByteBuilder
Here is a complete step by step guide on how to use it. It is on github.
Java Boon - Auto Growable Byte Buffer like a ByteBuilder
Have you ever wanted an easy to use buffer array that grow automatically and/or you can give it a fix size and just add stuff to it? I have. I wrote one too.
Look.. I can write strings to it (it converts them to UTF-8).
ByteBuf buf = new ByteBuf();
buf.add(bytes("0123456789\n"));
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456END\n");
Then later I can read the String out of the buffer:
String out = new String(buf.readAndReset(), 0, buf.len());
assertEquals(66, buf.len());
assertTrue(out.endsWith("END\n"));
I never have to set the size of the array. It will auto-grow as needed in an efficient manner.
If I know exactly how large my data is going to be than I can save some bounds checking by using createExact.
ByteBuf buf = ByteBuf.createExact(66);
buf.add(bytes("0123456789\n"));
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456END\n");
assertEquals(66, buf.len());
If I use create exact, then I am saying... hey.. I know exactly how big it can grow to and it will never go over this number and if it does...you can hit me over the head with a sack of rocks!
The following hits you over the head with a sack of rocks! THROWS AN EXCEPTION!!!!
ByteBuf buf = ByteBuf.createExact(22);
buf.add(bytes("0123456789\n"));
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456789\n");
buf.add("0123456END\n");
It works with doubles.
ByteBuf buf = ByteBuf.createExact(8);
//add the double
buf.add(10.0000000000001);
byte[] bytes = buf.readAndReset();
boolean worked = true;
worked |= idxDouble(bytes, 0) == 10.0000000000001 || die("Double worked");
It works with float.
ByteBuf buf = ByteBuf.createExact(8);
//add the float
buf.add(10.001f);
byte[] bytes = buf.readAndReset();
boolean worked = true;
worked |= buf.len() == 4 || die("Float worked");
//read the float
float flt = idxFloat(bytes, 0);
worked |= flt == 10.001f || die("Float worked");
It works with int.
ByteBuf buf = ByteBuf.createExact(8);
//Add the int to the array
buf.add(99);
byte[] bytes = buf.readAndReset();
boolean worked = true;
//Read the int back
int value = idxInt(bytes, 0);
worked |= buf.len() == 4 || die("Int worked length = 4");
worked |= value == 99 || die("Int worked value was 99");
It works with char.
ByteBuf buf = ByteBuf.createExact(8);
//Add the char to the array
buf.add('c');
byte[] bytes = buf.readAndReset();
boolean worked = true;
//Read the char back
int value = idxChar(bytes, 0);
worked |= buf.len() == 2 || die("char worked length = 4");
worked |= value == 'c' || die("char worked value was 'c'");
It works with short.
ByteBuf buf = ByteBuf.createExact(8);
//Add the short to the array
buf.add((short)77);
byte[] bytes = buf.readAndReset();
boolean worked = true;
//Read the short back
int value = idxShort(bytes, 0);
worked |= buf.len() == 2 || die("short worked length = 2");
worked |= value == 77 || die("short worked value was 77");
It even works with bytes.
ByteBuf buf = ByteBuf.createExact(8);
//Add the byte to the array
buf.add( (byte)33 );
byte[] bytes = buf.readAndReset();
boolean worked = true;
//Read the byte back
int value = idx(bytes, 0);
worked |= buf.len() == 1 || die("byte worked length = 1");
worked |= value == 33 || die("byte worked value was 33");
You can add all sorts of primitives to your byte array.
boolean worked = true;
ByteBuf buf = ByteBuf.create(1);
//Add the various to the array
buf.add( (byte) 1 );
buf.add( (short) 2 );
buf.add( (char) 3 );
buf.add( 4 );
buf.add( (float) 5 );
buf.add( (long) 6 );
buf.add( (double)7 );
worked |= buf.len() == 29 || die("length = 29");
byte[] bytes = buf.readAndReset();
byte myByte;
short myShort;
char myChar;
int myInt;
float myFloat;
long myLong;
double myDouble;
Now we just verify that we can read everything back.
myByte = idx ( bytes, 0 );
myShort = idxShort ( bytes, 1 );
myChar = idxChar ( bytes, 3 );
myInt = idxInt ( bytes, 5 );
myFloat = idxFloat ( bytes, 9 );
myLong = idxLong ( bytes, 13 );
myDouble = idxDouble ( bytes, 21 );
worked |= myByte == 1 || die("value was 1");
worked |= myShort == 2 || die("value was 2");
worked |= myChar == 3 || die("value was 3");
worked |= myInt == 4 || die("value was 4");
worked |= myFloat == 5 || die("value was 5");
worked |= myLong == 6 || die("value was 6");
worked |= myDouble == 7 || die("value was 7");
Once you call
byte[] bytes = buf.readAndReset()
then you are saying that you are done with the ByteBuffer!
Once you ask for the bytes, it becomes useless as it sets the internal byte array to nothing.
When you call readAndReset, it is giving you its buffer. Here is my internal state, you can have it, but I am going to set it to null so nobody else uses it.
It is ok. Just create another if you are sure only one instance at a time is using the buffer (byte []).
You can even use the buffer you were just using as in
ByteBuf buf2 = new ByteBuf.create(bytes);
This is because no buffer gets copied. ByteBuf writes to the buffer you give it. If you want another copy to be given to ByteBuf then do this:
ByteBuf buf2 = new ByteBuf.create( copy(bytes) );
This is boon after all. :)
Come check out boon. You get the above class and idx, and idxInt and idxLong for free!
Solution 4 - Java
Let's see. There is the ByteBuffer class in Java.
http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html
It has bulk methods that transfer contiguous sequences of bytes from a byte array to hardware buffers. It would do the trick.
It also has absolute and relative get and put methods that read and write byte[]s and other primitives to / for the byte buffer.
It also has methods for compacting, duplicating, and slicing a byte buffer.
// Creates an empty ByteBuffer with a 1024 byte capacity
ByteBuffer buf = ByteBuffer.allocate(1024);
// Get the buffer's capacity
int capacity = buf.capacity(); // 10
buf.put((byte)0xAA); // position=0
// Set the position
buf.position(500);
buf.put((byte)0xFF);
// Read the position 501
int pos = buf.position();
// Get remaining byte count
int remaining = buf.remaining(); (capacity - position)
It also has a bulk put to put an array, which is pretty close to the append you were asking for:
public final ByteBuffer put(byte[] src)
See: http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html#put(byte[])
I wrote my own little lib for manipulating byte arrays. :)
You can add them like so
byte [] a = ...
byte [] b = ...
byte [] c = ...
a = add(a, b);
a = add(a, c);
this would give you all of the contents of b, and c after the contents for a.
If you wanted to grow a by 21, you could do the following:
a = grow( letters, 21);
If you wanted to double the size of a, you could do the following:
a = grow( letters, 21);
See...
https://github.com/RichardHightower/boon/blob/master/src/main/java/org/boon/core/primitive/Byt.java
byte[] letters =
arrayOfByte(500);
assertEquals(
500,
len(letters)
);
Create
byte[] letters =
array((byte)0, (byte)1, (byte)2, (byte)3);
assertEquals(
4,
len(letters)
);
Index
byte[] letters =
array((byte)'a', (byte)'b', (byte)'c', (byte)'d');
assertEquals(
'a',
idx(letters, 0)
);
assertEquals(
'd',
idx(letters, -1)
);
assertEquals(
'd',
idx(letters, letters.length - 1)
);
idx(letters, 1, (byte)'z');
assertEquals(
(byte)'z',
idx(letters, 1)
);
Contains
byte[] letters =
array((byte)'a',(byte) 'b', (byte)'c', (byte)'d');
assertTrue(
in((byte)'a', letters)
);
assertFalse(
in((byte)'z', letters)
);
Slice:
byte[] letters =
array((byte)'a', (byte)'b', (byte)'c', (byte)'d');
assertArrayEquals(
array((byte)'a', (byte)'b'),
slc(letters, 0, 2)
);
assertArrayEquals(
array((byte)'b', (byte)'c'),
slc(letters, 1, -1)
);
//>>> letters[2:]
//['c', 'd']
//>>> letters[-2:]
//['c', 'd']
assertArrayEquals(
array((byte)'c', (byte)'d'),
slc(letters, -2)
);
assertArrayEquals(
array((byte)'c', (byte)'d'),
slc(letters, 2)
);
//>>> letters[:-2]
// ['a', 'b']
assertArrayEquals(
array((byte)'a', (byte)'b'),
slcEnd(letters, -2)
);
//>>> letters[:-2]
// ['a', 'b']
assertArrayEquals(
array((byte)'a',(byte) 'b'),
slcEnd(letters, 2)
);
Grow
byte[] letters =
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e');
letters = grow( letters, 21);
assertEquals(
'e',
idx(letters, 4)
);
assertEquals(
'a',
idx(letters, 0)
);
assertEquals(
len(letters),
26
);
assertEquals(
'\0',
idx(letters, 20)
);
Shrink:
letters = shrink ( letters, 23 );
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c'),
letters
);
Copy:
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'),
copy(array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'))
);
Add:
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f'),
add(array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e'), (byte)'f') );
The add actually adds them together by using System.arraycopy (considering Unsafe, but not yet).
Add one array to another:
assertArrayEquals(
array( (byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f'),
add( array((byte)'a', (byte)'b', (byte)'c', (byte)'d'), array((byte)'e', (byte)'f') )
);
Insert:
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
insert( array((byte)'a', (byte)'b', (byte)'d', (byte)'e', (byte)'f', (byte)'g'), 2, (byte)'c' )
);
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
insert( array((byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'), 0, (byte)'a' )
);
assertArrayEquals(
array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g'),
insert( array((byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'g'), 5, (byte)'f' )
);
Here is a peek at a few of the methods:
public static byte[] grow(byte [] array, final int size) {
Objects.requireNonNull(array);
byte [] newArray = new byte[array.length + size];
System.arraycopy(array, 0, newArray, 0, array.length);
return newArray;
}
public static byte[] grow(byte [] array) {
Objects.requireNonNull(array);
byte [] newArray = new byte[array.length *2];
System.arraycopy(array, 0, newArray, 0, array.length);
return newArray;
}
public static byte[] shrink(byte[] array, int size) {
Objects.requireNonNull(array);
byte[] newArray = new byte[array.length - size];
System.arraycopy(array, 0, newArray, 0, array.length-size);
return newArray;
}
public static byte[] copy(byte[] array) {
Objects.requireNonNull(array);
byte[] newArray = new byte[array.length];
System.arraycopy(array, 0, newArray, 0, array.length);
return newArray;
}
public static byte[] add(byte[] array, byte v) {
Objects.requireNonNull(array);
byte[] newArray = new byte[array.length + 1];
System.arraycopy(array, 0, newArray, 0, array.length);
newArray[array.length] = v;
return newArray;
}
public static byte[] add(byte[] array, byte[] array2) {
Objects.requireNonNull(array);
byte[] newArray = new byte[array.length + array2.length];
System.arraycopy(array, 0, newArray, 0, array.length);
System.arraycopy(array2, 0, newArray, array.length, array2.length);
return newArray;
}
public static byte[] insert(final byte[] array, final int idx, final byte v) {
Objects.requireNonNull(array);
if (idx >= array.length) {
return add(array, v);
}
final int index = calculateIndex(array, idx);
//Object newArray = Array.newInstance(array.getClass().getComponentType(), array.length+1);
byte [] newArray = new byte[array.length+1];
if (index != 0) {
/* Copy up to the location in the array before the index. */
/* src sbegin dst dbegin length of copy */
System.arraycopy( array, 0, newArray, 0, index );
}
boolean lastIndex = index == array.length -1;
int remainingIndex = array.length - index;
if (lastIndex ) {
/* Copy the area after the insert. Make sure we don't write over the end. */
/* src sbegin dst dbegin length of copy */
System.arraycopy(array, index, newArray, index + 1, remainingIndex );
} else {
/* Copy the area after the insert. */
/* src sbegin dst dbegin length of copy */
System.arraycopy(array, index, newArray, index + 1, remainingIndex );
}
newArray[index] = v;
return newArray;
}
public static byte[] insert(final byte[] array, final int fromIndex, final byte[] values) {
Objects.requireNonNull(array);
if (fromIndex >= array.length) {
return add(array, values);
}
final int index = calculateIndex(array, fromIndex);
//Object newArray = Array.newInstance(array.getClass().getComponentType(), array.length+1);
byte [] newArray = new byte[array.length + values.length];
if (index != 0) {
/* Copy up to the location in the array before the index. */
/* src sbegin dst dbegin length of copy */
System.arraycopy( array, 0, newArray, 0, index );
}
boolean lastIndex = index == array.length -1;
int toIndex = index + values.length;
int remainingIndex = newArray.length - toIndex;
if (lastIndex ) {
/* Copy the area after the insert. Make sure we don't write over the end. */
/* src sbegin dst dbegin length of copy */
System.arraycopy(array, index, newArray, index + values.length, remainingIndex );
} else {
/* Copy the area after the insert. */
/* src sbegin dst dbegin length of copy */
System.arraycopy(array, index, newArray, index + values.length, remainingIndex );
}
for (int i = index, j=0; i < toIndex; i++, j++) {
newArray[ i ] = values[ j ];
}
return newArray;
}
More....