Need complex bash text cropping

Question

There is a sql dump of the test chat, it looks like this (I quote 3 lines ... the formatting is curved, so it seems that there are 6 of them .. visually):

INSERT INTO "TEST-MSG" VALUES(840,1,263,'#user1/$user2;1633cb6c61f369e6','user1','user1'X'6512864C7028DCB17F5DDEA3663CE51082BF461A5EEA2E5C1024D47F82172914'1439889518,61,2,0NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 1)'NULLNULL3,4,1NULLNULL,3304554754,831352659);

INSERT INTO "TEST-MSG" VALUES(841,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'6233268E7FEFB4773EE63E79E5FAD5C2290C069273C8B60AA734C2435CF7F542'1439889536,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 2'NULLNULL3,4,1NULLNULL,585002415,153024551);

INSERT INTO "TEST-MSG" VALUES(842,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'81CADA68E5FF5BF22A5CDD01E33C061BDB7B68B7BF16D2C7602AAF17D2ABB922'1439889579,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 3'NULLNULL3,4,1NULLNULL,1544146789,153024552);

How can I cut all such lines in a loop so that the output in the text file remains only:

#user1/$user2; ТЕСТОВОЕ СООБЩЕНИЕ 1

#user1/$user2; ТЕСТОВОЕ СООБЩЕНИЕ 2

#user1/$user2; ТЕСТОВОЕ СООБЩЕНИЕ 3

Most likely you need a combination of sed, grep and awk.

bash is slow, if you need to quickly, it is better to write a text parser on perl using regular expressions.
If you write a variant on Pearl, I will be very grateful to you) I myself, unfortunately, do not know him ... therefore, bash)
I can not say that I know Perl, but nonetheless I write with "directories" sometimes not very simple scripts.
I can jot down on a pearl, only 2 questions: 1) are # and $ required in #user1/$user2 and 2) are NULLNULL before and after the message.
The grid and the dollar have no special meaning - you can safely cut them) I checked a couple of hundred lines - indeed, all messages are framed in 610NULNUL ... NULNUL3

Answer 1 · 2015-11-03T09:43:26

 use v5.10; use strict; use warnings; open(my $IN, '<', $ARGV[0]) or die "Cannot open file $ARGV[0]"; while (<$IN>) { if (/(#\w+\/\$\w+);.+NULLNULL,\'(.+)\'NULLNULL/g) { print "$1; $2\n"; } } close($IN);

Displays what looks like ... (# letters numbers / $ letters numbers); ... NULLNULL '(message body)' NULLNULL ...

If you throw lines with open and close and leave while (<>) { , will read STDIN, you can make cat filename | perl script.pl cat filename | perl script.pl

If you fix it like this:

  if (/#(\w+)\/\$(\w+);.+NULLNULL,\'(.+)\'NULLNULL/g) { print "$1 $2 '$2'\n"; }

that will output

 user1 user2 'тело сообщения'

(but this is not checked)

 #!/usr/bin/perl while (<>) { print "$1 $2 $3 $4 '$5'\n" if (/#(\w+)\/\$(\w+);[^']+\',\'(\w+)\',\'(\w+).+NULLNULL,\'(.+)\'NULLNULL/g) }

Try something like this, should display user1 user2 user3 user4 'message' , but did not check. UPD checked works. Put in the file xxx.pl , make it executable, run as cat <исходный файл> | ./xxx.pl cat <исходный файл> | ./xxx.pl If you only need user3 or user4, remove $ 4 or $ 3, respectively, P.S. You can remove all backslashes before single quotes, I used them in the process of debugging.

will not work if there is no lattice and dollar, if they are not always present, give, plz, an example of a line without them.
I wouldn’t focus on his NULLs, it’s better to bet on commas and on single quotes.
Run perl <имя_файла_со_скриптом> <имя_файла_с_текстом>
perl <имя_файла_со_скриптом> <имя_файла_с_текстом>
@ andy.37 On Linux, it is enough to allow the file containing the script to execute chmod a+x имя_файла and enter #!/bin/perl

Qwertiy ♦ Qwertiy 76.2k 17 73 200 · Answer 2 · 2015-11-03T11:12:10

Here are the regulars needed:

 document.querySelector('output').textContent = document.querySelector('pre').textContent .replace(/^INSERT INTO "TEST-MSG" VALUES\(\d+,\d+,\d+,'((?:''|[^';])+)(?:''|[^'])+'(?:,'(?:''|[^'])+'){2}X'(?:''|[^'])+'(?:\w+,){2,4}'((?:''|[^'])*)'\w+(?:,\w+){4}\);$/gim, "$1; $2")

 pre, output { white-space: pre; font-family: monospace; }

 <pre>INSERT INTO "TEST-MSG" VALUES(840,1,263,'#user1/$user2;1633cb6c61f369e6','user1','user1'X'6512864C7028DCB17F5DDEA3663CE51082BF461A5EEA2E5C1024D47F82172914'1439889518,61,2,0NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 1)'NULLNULL3,4,1NULLNULL,3304554754,831352659); INSERT INTO "TEST-MSG" VALUES(841,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'6233268E7FEFB4773EE63E79E5FAD5C2290C069273C8B60AA734C2435CF7F542'1439889536,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 2'NULLNULL3,4,1NULLNULL,585002415,153024551); INSERT INTO "TEST-MSG" VALUES(842,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'81CADA68E5FF5BF22A5CDD01E33C061BDB7B68B7BF16D2C7602AAF17D2ABB922'1439889579,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 3'NULLNULL3,4,1NULLNULL,1544146789,153024552);</pre> <output></output> (841,1,263, '# user1 / $ user2; 1633cb6c61f369e6', 'user2', 'user2'X'6233268E7FEFB4773EE63E79E5FAD5C2290C069273C8B60AA734C2435CF7F542'1439889536,610NULLNULL,' <pre>INSERT INTO "TEST-MSG" VALUES(840,1,263,'#user1/$user2;1633cb6c61f369e6','user1','user1'X'6512864C7028DCB17F5DDEA3663CE51082BF461A5EEA2E5C1024D47F82172914'1439889518,61,2,0NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 1)'NULLNULL3,4,1NULLNULL,3304554754,831352659); INSERT INTO "TEST-MSG" VALUES(841,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'6233268E7FEFB4773EE63E79E5FAD5C2290C069273C8B60AA734C2435CF7F542'1439889536,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 2'NULLNULL3,4,1NULLNULL,585002415,153024551); INSERT INTO "TEST-MSG" VALUES(842,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'81CADA68E5FF5BF22A5CDD01E33C061BDB7B68B7BF16D2C7602AAF17D2ABB922'1439889579,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 3'NULLNULL3,4,1NULLNULL,1544146789,153024552);</pre> <output></output> (842,1,263, '# user1 / $ user2; 1633cb6c61f369e6', 'user2', 'user2'X'81CADA68E5FF5BF22A5CDD01E33C061BDB7B68B7BF16D2C7602AAF17D2ABB922'1439889579,610NULLNULL,' <pre>INSERT INTO "TEST-MSG" VALUES(840,1,263,'#user1/$user2;1633cb6c61f369e6','user1','user1'X'6512864C7028DCB17F5DDEA3663CE51082BF461A5EEA2E5C1024D47F82172914'1439889518,61,2,0NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 1)'NULLNULL3,4,1NULLNULL,3304554754,831352659); INSERT INTO "TEST-MSG" VALUES(841,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'6233268E7FEFB4773EE63E79E5FAD5C2290C069273C8B60AA734C2435CF7F542'1439889536,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 2'NULLNULL3,4,1NULLNULL,585002415,153024551); INSERT INTO "TEST-MSG" VALUES(842,1,263,'#user1/$user2;1633cb6c61f369e6','user2','user2'X'81CADA68E5FF5BF22A5CDD01E33C061BDB7B68B7BF16D2C7602AAF17D2ABB922'1439889579,610NULLNULL,'ТЕСТОВОЕ СООБЩЕНИЕ 3'NULLNULL3,4,1NULLNULL,1544146789,153024552);</pre> <output></output>

Need complex bash text cropping

2 answers 2

More articles: