random bytes format python

[kopev]

hello,
I would like to know I always understanding that bytes in binary format start with the symbol "\" (antislash) but when I generate random bytes with python sometimes it give me strange result look above :

>>> import os
>>> os.urandom(8)
b'<\x8b\x0feH{\x99\x01'
>>> os.urandom(8)
b'#\x93\xebg\xf2\xbee\xd6'
>>> os.urandom(8)
b'\xa4\x14\x0e\xd4\x9a\xba:2'
>>> os.urandom(8)
b'\xc5\x98\x90\x9dE\x17x\xa3'
>>> os.urandom(8)
b'\xb9m\xd4R\xf1\xfa\xee\x93'
>>> os.urandom(8)
b'(Fm\xcb}\xbd\xb7\x83'
>>> os.urandom(8)
b'\xf5b\x8d\x9a\x12>\x8bb'

some byte start with "\" and some with # or even ( ??? I totally don't get it if someone can explain

thanks

[Chervil]

The prefix \x means a hexadecimal value follows. (two hex digits per character)
It is used for non-printable characters.

example,
b'<\x8b\x0feH{\x99\x01'

contains eight characters as follows:

      1     2      3     4     5     6     7       8
b'    <    \x8b  \x0f    e     H     {    \x99    \x01    '

See Escape Sequence
https://docs.python.org/2.0/ref/strings.html

[kopev]

then why # and ( are not stored in hex like other bytes ?

[Chervil]

# and ( are printable characters. Put simply anything in the ASCII range of 32 to 126 is printable, outside of that range, hex would be used.
https://ascii.cl/

# is ASCII 35
( is ASCII 40

Those are in the range 32 to 126 and therefore shown as ordinary characters.

[kopev]

look at that :

I generate 7 random bytes
>>> os.urandom(7)
b'\x06\xcf\x10^$\xa8\xab'
>>>

after that I add on my C code :
unsigned char test[7] = "\x06\xcf\x10^$\xa8\xab";

and it don't work !!

my editor say that this sequence is 8 bytes I don't understand !!

edit : I think cplusplus.com BDD was overloaded i havent your message in time, why then python print them ??

[kopev]

should I store as hex array ?

[Peter87]

kopev wrote:

my editor say that this sequence is 8 bytes I don't understand !!

One extra byte is needed to store the null character '\0' that marks the end of the string.

[Chervil]

A string enclosed in double quotes such as "dog" is actually a sequence of characters ending with a null character (a byte with the value 0). Thus "dog" requires 4 characters.

 char word[4] = { 'd', 'o', 'g', 0  };

You could just make the array one byte larger to allow for that null.

    unsigned char test[9] = "\x06\xcf\x10^$\xa8\xab\x8c";

Or maybe try something like this - which is quite long and awkward

 unsigned char test[8] = { '\x06', '\xcf', '\x10', '^', '$', '\xa8', '\xab', '\x8c' };

You might even find you need this in c++

 unsigned char test[8] = { u'\x06',  u'\xcf', u'\x10', '^', '$', u'\xa8', u'\xab', u'\x8c' };

[kopev]

unsigned char test[7] = "XV\x95\xec\x1f0\x8b";

I don't understand why you are talking about (sorry) because the code here don't have any error

but this one : unsigned char test[7] = "\x06\xcf\x10^$\xa8\xab";
have error
however they are same size
-----------------------------------------------------------------------------
char array[] = "abc";

// is equal to

char array[] = "\x61\x62\x63";

example taken from https://www.linuxquestions.org/questions/programming-9/how-to-define-hex-character-array-in-c-668213/

why I would like the character null at the end of my hex array if i know the size

[kopev]

this is most likely because a character got misinterpreted or i should escape it or something
however i'm new to C i don't know too much about problem like that

[Peter87]

You might know how long the string is but functions such as strcpy, strlen and printf doesn't. They use the null character to detect that the end of the string has been reached.

[Chervil]

Let's look at some of the ways that strings work.
Try this code:

#include <stdio.h>

int main()
{    
    char array[] = "abc";
    
    printf("Size of array = %d\n", sizeof(array)  );


    for (unsigned int i=0; i<sizeof(array); ++i)
        printf("%2d    %c   %02x \n", i+1, array[i], array[i]);
}

This is the output:

Size of array = 4
 1    a   61
 2    b   62
 3    c   63
 4        00

By all means look on other internet sites for information. But not everything posted on a forum may be correct. Usually stackoverflow can useful.

This code gave a compiler warning when I tried it:

    unsigned char test[7] = "XV\x95\xec\x1f0\x8b";

[Warning] hex escape sequence out of range [enabled by default]

Putting that into the sample program shown above, this is the output, although the length is 7, there are only six meaningful characters, followed by a null.

Size of array = 7
 1    X   58
 2    V   56
 3    ò   95
 4    ý   ec
 5    ­   f0
 6    ï   8b
 7        00

Note that the sequence \x1f0 is truncated to just \xf0.

Presumably it should have been x1f followed by an ordinary character '0'. One solution is to put \x1f\x30 where \x30 is the hex representation of the character '0'.

This takes us to here:

    unsigned char array[] = "XV\x95\xec\x1f\x30\x8b";

and corresponding output:

Size of array = 8
 1    X   58
 2    V   56
 3    ò   95
 4    ý   ec
 5    ▼   1f
 6    0   30
 7    ï   8b
 8        00

Note the size is now 8.

[kopev]

I dont fucking care about string or str function or anything
I don't think I'm forced to put a null after my buffer of bytes if I know the size

I want a bytes buffer without null terminated

stop talking me about string please

I will use unsigned char * now even if it add a null char but i don't know what the point of using C/C++ if it add a null for you, fuck it...

[kopev]

https://stackoverflow.com/questions/2420780/how-to-assign-a-value-to-a-char-using-hex-notation

Although the resulting executable will have the same behavior these are not all the same: 0x63 is an int literal that will be converted to a char. Some compilers will issue a 'truncation' warning for this. char cat[] = { '\x63', '\x61', '\x74', '\x00' }; will be the same as char cat[] = "\x63\x61\x74"; and not generate a compiler warning. – Bob Reynolds Apr 13 '13 at 0:51

extern const unsigned char* test[8 + 1] = "\x63\x61\x74\x63\x61\x74\x63\x61";

this give me error but this is same from the stackexchange example lol...

initialization with '{...}' expected for aggregate object

[kopev]

I would like a simple format like "\x80\x80\x80" because i will need to modify with python the value you know

I dont know if my example above will work and will be interpreted as hex

[Chervil]

I'm not totally sure what the end goal is.

However, it you are using python to generate values to be used in a C program, I can think of two approaches.
1. Using python, write the data to a file. Then later the C program can read from that file.

2. Use python to generate a file containing C source code which can be #included in a C program.

Python code follows:
1. Simply write the data to a (binary) file.

import os

print ("Testing Urandom\n")

size = 7
data = os.urandom(size)

with open('array.bin', 'wb') as f:
    f.write(data)

2. Write C source code to a header file:

import os

print ("Testing Urandom\n")

size = 7
data = os.urandom(size)

with open('array.h', 'w') as f:
    first = 1
    column = 0;
    f.write("    unsigned char array[")
    f.write(str(size))
    f.write("] = { ")
    for c in data:

#       hex_string = '\\x{:02x}'.format(ord(c))
        hex_string = '\\x{:02x}'.format(c)
        
        if (first):
            first = 0
        else:
            f.write(", ")

        column = column + 1
        if (column > 8):
            column = 1
            f.write("\n        ")
            
        f.write("'" + hex_string + "'")

    f.write(" };\n")

Resulting output in file 'array.h'

    unsigned char array[7] = { '\x66', '\x5c', '\xfd', '\xdf', '\x70', '\xfa', '\x3e' };

Note, I have very limited knowledge of python, so there may be errors or better ways to do this.

[Peter87]

kopev wrote:

Although the resulting executable will have the same behavior these are not all the same: 0x63 is an int literal that will be converted to a char. Some compilers will issue a 'truncation' warning for this.

Are you sure? Sounds like a stupid compiler. It shouldn't be that hard for a compiler to figure out that the value is small enough to fit in an unsigned char. If you happen to use such a compiler you can probably turn off this warning anyway. Note that it's not a narrowing conversion so the code is perfectly valid C++ code.