Again Cyrillic entries error

[rumen]

I got again Cyrillic entries in Bakery 1.83 in the order form error but this time the problem is not in the DB table collation and not in the template ... so what I miss?

Rumen

[jacobi22]

Bakery is prepared for Latin characters only

here the form-check from save_form.php for the company field

Code: [Select]

if (!preg_match('#^[\p{Latin}'.$us.'0-9.+&\s\-]{0,50}$#u', $value)) {
you can try it with this

Code: [Select]

if (!preg_match('#^[\p{Cyrillic}+|\p{Latin}'.$us.'0-9.+&\s\-]{0,50}$#u', $value)) {but not sure, not possible, to test it for me at the moment

P.S.: there are a lot of field checks for nearly every field, be very carefull with this and try to understand first,  how the preg_match works in every check
for example:
country - allowed only two letters A-Z

Code: [Select]

if (!preg_match('#^[A-Z]{2}$#u', $value)) {
maybe it works with

Code: [Select]

if (!preg_match (#^[a-zA-Z\p{Cyrillic}\d\s\-]+$/u', $value)) {
or with this for Latin Users also

Code: [Select]

if (!preg_match('#^[\p{Cyrillic}+|\p{Latin}'.$us.'0-9.+&\s\-]{2}$#u', $value)) {
go step by step and check the form after every chenges in the field check

[rumen]

I have changed the fields, but got same error. I remember I got the same error before, but it was in ver. 1.77. Now this is ver. 1.83, otherwise I would copy the files from the other one that works

[rumen]

In this file save_form.php I see there is Cyrillic support.

[jacobi22]

plaese change this code for a test to

Code: [Select]

if (!empty($MOD_BAKERY['ADD_CHARSET'])) {
    switch (strtolower($MOD_BAKERY['ADD_CHARSET'])) {
        case 'cyrillic':
//            $us = '\p{Cyrillic}';
            $us = '+|\p{Cyrillic}';
            break;
        case 'greek':
//            $us = '\p{Greek}';
            $us = '+|\p{Greek}';
            break;
        case 'hebrew':
//            $us = '\p{Hebrew}';
            $us = '+|\p{Hebrew}';
            break;
        case 'arabic':
//            $us = '\p{Arabic}';
            $us = '+|\p{Arabic}';
            break;
    }
}
works for me with a regex-tester

[rumen]

I have change it, but I got the same error   what can be the reason?

[jacobi22]

There is a logical problem inside of the proof methods, i need some minutes to repair this, but i have to check every method first

[jacobi22]

so, next try  (works for me with latin, cyrillic, hebrew, arabic) -
the charset like this come's from the translation file - in this case specially from the RU.php. All Pages from the Latin languages has the default charset "latin" and no specially definition in the language files.
it's very important, that the translator of the language file set this charset in the language file, otherwhise, he get latin as charset, so check also your bulgarian language file for that
it's also important, that the visited page has this selected language (see page settings). For example: a page with language = english can in this form only work with latin characters, but not with cyrillic
but the check for latin chars work in every language, that means: you can use also latin chars on a russian or arabic page

i hope, it works for you also

$us = '';
if (!empty($MOD_BAKERY['ADD_CHARSET'])) {
    switch (strtolower($MOD_BAKERY['ADD_CHARSET'])) {
        case 'cyrillic':
            $us = '\p{Cyrillic}';
            break;
        case 'greek':
            $us = '\p{Greek}';
            break;
        case 'hebrew':
            $us = '\p{Hebrew}';
            break;
        case 'arabic':
            $us = '\p{Arabic}';
            break;
        default:
            $us = '\p{Latin}';
            break;
    }
}

// Check the textfields
foreach ($_POST as $field => $value) {
    if ($field != 'pay_methods') {
        $field = strip_tags($field);
        $value = strip_tags($value);

        if (strpos($field, 'company') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9.+&\s\-]{1,50}+$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_NAME'];
            }
        }

        if (strpos($field, 'first_name') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9\s\-]{1,50}+$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . '  ' . $MOD_BAKERY['ERR_INVAL_NAME'];
            }
        }

        if (strpos($field, 'last_name') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9\s\-]{1,50}+$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_NAME'];
            }else{echo "geht doch";}
        }

        if (strpos($field, 'cust_tax_no') !== false &&
                strpos($setting_tax_group, $setting_shop_country) !== false) {
            include('check_vat.php');
            $value = trim($value);
            if (!check_vat($value, $setting_tax_group)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_CUST_TAX_NO'];
            }
        }

        if (strpos($field, 'street') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9.+&\s\-]{1,50}$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_STREET'];
            }
        }

        if (strpos($field, 'city') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9.+&\s\-]{1,50}$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_CITY'];
            }
        }

        if (strpos($field, 'state') !== false) {
            if (!preg_match('/^[a-zA-Z'. $us .'0-9.+&\s\-]{1,50}$/u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_STATE'];
            }
        }

        if (strpos($field, 'country') !== false) {
            if (!preg_match('#^[A-Z]{2}$#u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_COUNTRY'];
            }
        }

        if (strpos($field, 'email') !== false) {
            if (!preg_match('#^.+@.+\..+$#u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_EMAIL'];
            }
        }

        if (strpos($field, 'zip') !== false) {
            if (!preg_match('#^[A-Za-z0-9\s\-]{4,10}$#u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_ZIP'];
            }
        }

        if (strpos($field, 'phone') !== false) {
            if (!preg_match('#^[0-9)(xX+./\s\-]{7,20}$#u', $value)) {
                $error_bg[] = $field;
                $errors[] = htmlspecialchars($value, ENT_QUOTES) . ' ' . $MOD_BAKERY['ERR_INVAL_PHONE'];
            }
        }

        $$field = strip_tags(trim($value));
    }
}

[rumen]

I have changed the code, but same error appeared.

[jacobi22]

what can i say... it works for me with some fictive google translations in the fields -> https://gyazo.com/2ee4eb5d54f18afdd4abfc9e2265771d

is it russian or bulgarian?
you have this line in the bakery-language file?

Code: [Select]

$MOD_BAKERY['ADD_CHARSET'] = 'Cyrillic';
one line from the code has too much content, in the area for the last name at the end, remove this

else{echo "geht doch";}

but it has nothing to do with the form check

Have you try your version from the 1.73?

[rumen]

I took the language file from another site with Bakery which works. But there is no such raw there. I put the raw there and now it works (just should not have , and . in the fields) but works and again the same problem with the mail confirmation like before .... both e-mails not readable:

Code: [Select]

Здравейте Румен Запрянов

Благодарим Ви за поръчката от El Tempo.
Ето детайли за вашата поръчка:

Номер: 0002
Име: Пилешка супа
Хляб: Без хляб
Количество: 1
Цена: BGN 2.90
Доставка: BGN 1.00
Сума: BGN 2.90

-------------------------------------
Междинен сбор: BGN 2.90
Цена за доставката: BGN 1.00
Inclusive 0.0% Данък: BGN 0.00
-------------------------------------
-------------------------------------
Общо: BGN 3.90
-------------------------------------

Поръчката ще бъде доставена на адрес:

Румен Запрянов
Бусманци ул Циклама 2
1520 София
София


+359876415004


Вие избрахте да платите при доставката на куриера.


Благодарим Ви за доверието в нас.

Поздрави,
El Tempo

[rumen]

Like here  https://forum.WebsiteBaker.org/index.php/topic,30597.msg213299.html#msg213299  ,    but now all DataBase tables are UTF8 for sure.

[rumen]

Is there a difference for WB if some of the tables are UTF8_unicode_ci other tables are UTF8_general_ci? Because they have difference!

[jacobi22]

just should not have , and . in the fields

you need comma and the point? - If YES, which kind of field

again the same problem with the mail confirmation like before .... both e-mails not readable:

jau, thats not utf8, thats htmlspecialchars
was the problem solved in last time?

can you make a export from the database table mod_bakery_customer - i need only one complete order from there in sql-format.
i'll look to the way from the invoice from the buy-button into the database and from there to the email and the bakery administration

for a first try, go to view_confirmation.p hp  ~ line 160ff

Original-Code

Code: [Select]

// Make email headers
            if (defined('DEFAULT_CHARSET')) {
                $charset = DEFAULT_CHARSET;
            } else {
                $charset = 'utf-8';
            }
change it to

Code: [Select]

$charset = 'utf-8';
it's not longer needed to use if/else here, because there is only one charset possible, and this is utf8

i did not found any code to change the content from the invoice on the way into the email, so i think, you have the htmlspecialchars also in the database, but i have to build a russian bakery page with russion items and descriptions, to check this

Is there a difference for WB if some of the tables are UTF8_unicode_ci other tables are UTF8_general_ci?

yes, a small different, but it doesn't matter, what you use. both works
_unicode_ci is the default setting for WB and _general_ci is the default setting in your database. if some module not submit this character set, the database use the selected charset & collation

Some description from Stackoverflow

For any Unicode character set, operations performed using the _general_ci collation are faster than those for the _unicode_ci collation. For example, comparisons for the utf8_general_ci collation are faster, but slightly less correct, than comparisons for utf8_unicode_ci. The reason for this is that utf8_unicode_ci supports mappings such as expansions; that is, when one character compares as equal to combinations of other characters. For example, in German and some other languages “ß” is equal to “ss”. utf8_unicode_ci also supports contractions and ignorable characters. utf8_general_ci is a legacy collation that does not support expansions, contractions, or ignorable characters. It can make only one-to-one comparisons between characters.

you can also use utf8mb4_unicode_ci

change the line in your config.php like this

Code: [Select]

define('DB_CHARSET',      'utf8mb4_unicode_ci');
and start the upgrade-script. there is no risk, if you have all table with one of the utf8-characters like utf8mb4_unicode_ci, utf8mb4_general_ci, utf8_unicode_ci or utf8_general_ci

if you use emoticons in ckeditor, utf8mb4_......_ci is neccesary

[rumen]

I have changed the code with:

Code: [Select]

$charset = 'utf-8';
But no effect at all, same error.

I sent you by mail the data.

[rumen]

So the issue was solved!!!

I just took the EN.php language file, we input the line for Cyrillic there and I made new translation of the file. Now works fine!

[rumen]

Special thanks to jacobi22 for the help. THanks again brother!

[jacobi22]

additional informations for everybody:

it's very important, that the language files are converted to utf8 without boom
Use Notepad++ or other semi-proffessional editors, to edit in this files or to convert it