Having looked at quite a few solutions here with various degrees of understanding, I wondered what to do if the string was comma separated…
the problem
While trying to process a csv of contact information, I needed a solution this problem: trim extraneous whitespace and some junk, but preserve trailing commas, and internal whitespace. Working with a field containing notes on the contacts, I wanted to remove the garbage, leaving the good stuff. Trimming out all the punctuation and chaff, I didn’t want to lose the whitespace between compound tokens as I didn’t want to rebuild later.
regex and patterns: [\s_]+?\W+
The pattern looks for single instances of any whitespace character and the underscore (‘_’) from 1 to an unlimited number of times lazily (as few characters as possible) with [\s_]+?
that come before non-word characters occurring from 1 to an unlimited amount of time with this: \W+
(is equivalent to [^a-zA-Z0-9_]
). Specifically, this finds swaths of whitespace: null characters (\0), tabs (\t), newlines (\n), feed-forward (\f), carriage returns (\r).
I see the advantage to this as two-fold:
that it doesn’t remove whitespace between the complete words/tokens that you might want to keep together;
Python’s built in string method strip()
doesn’t deal inside the string, just the left and right ends, and default arg is null characters (see below example: several newlines are in the text, and strip()
does not remove them all while the regex pattern does). text.strip(' \n\t\r')
This goes beyond the OPs question, but I think there are plenty of cases where we might have odd, pathological instances within the text data, as I did (some how the escape characters ended up in some of the text). Moreover, in list-like strings, we don’t want to eliminate the delimiter unless the delimiter separates two whitespace characters or some non-word character, like ‘-,’ or ‘-, ,,,’.
NB: Not talking about the delimiter of the CSV itself. Only of instances within the CSV where the data is list-like, ie is a c.s. string of substrings.
Full disclosure: I’ve only been manipulating text for about a month, and regex only the last two weeks, so I’m sure there are some nuances I’m missing. That said, for smaller collections of strings (mine are in a dataframe of 12,000 rows and 40 odd columns), as a final step after a pass for removal of extraneous characters, this works exceptionally well, especially if you introduce some additional whitespace where you want to separate text joined by a non-word character, but don’t want to add whitespace where there was none before.
An example:
import re
text = "\"portfolio, derp, hello-world, hello-, -world, founders, mentors, :, ?, %, ,>, , ffib, biff, 1, 12.18.02, 12, 2013, 9874890288, .., ..., ...., , ff, series a, exit, general mailing, fr, , , ,, co founder, pitch_at_palace, ba, _slkdjfl_bf, sdf_jlk, )_(, jim.somedude@blahblah.com, ,dd invites,subscribed,, master, , , , dd invites,subscribed, , , , \r, , \0, ff dd \n invites, subscribed, , , , , alumni spring 2012 deck: https: www.dropbox.com s, \n i69rpofhfsp9t7c practice 20ignition - 20june \t\n .2134.pdf 2109 \n\n\n\nklkjsdf\""
print(f"Here is the text as formatted:\n{text}\n")
print()
print("Trimming both the whitespaces and the non-word characters that follow them.")
print()
trim_ws_punctn = re.compile(r'[\s_]+?\W+')
clean_text = trim_ws_punctn.sub(' ', text)
print(clean_text)
print()
print("what about 'strip()'?")
print(f"Here is the text, formatted as is:\n{text}\n")
clean_text = text.strip(' \n\t\r') # strip out whitespace?
print()
print(f"Here is the text, formatted as is:\n{clean_text}\n")
print()
print("Are 'text' and 'clean_text' unchanged?")
print(clean_text == text)
This outputs:
Here is the text as formatted:
"portfolio, derp, hello-world, hello-, -world, founders, mentors, :, ?, %, ,>, , ffib, biff, 1, 12.18.02, 12, 2013, 9874890288, .., ..., ...., , ff, series a, exit, general mailing, fr, , , ,, co founder, pitch_at_palace, ba, _slkdjfl_bf, sdf_jlk, )_(, jim.somedude@blahblah.com, ,dd invites,subscribed,, master, , , , dd invites,subscribed, ,, , , ff dd
invites, subscribed, , , , , alumni spring 2012 deck: https: www.dropbox.com s,
i69rpofhfsp9t7c practice 20ignition - 20june
.2134.pdf 2109
klkjsdf"
using regex to trim both the whitespaces and the non-word characters that follow them.
"portfolio, derp, hello-world, hello-, world, founders, mentors, ffib, biff, 1, 12.18.02, 12, 2013, 9874890288, ff, series a, exit, general mailing, fr, co founder, pitch_at_palace, ba, _slkdjfl_bf, sdf_jlk, jim.somedude@blahblah.com, dd invites,subscribed,, master, dd invites,subscribed, ff dd invites, subscribed, alumni spring 2012 deck: https: www.dropbox.com s, i69rpofhfsp9t7c practice 20ignition 20june 2134.pdf 2109 klkjsdf"
Very nice.
What about 'strip()'?
Here is the text, formatted as is:
"portfolio, derp, hello-world, hello-, -world, founders, mentors, :, ?, %, ,>, , ffib, biff, 1, 12.18.02, 12, 2013, 9874890288, .., ..., ...., , ff, series a, exit, general mailing, fr, , , ,, co founder, pitch_at_palace, ba, _slkdjfl_bf, sdf_jlk, )_(, jim.somedude@blahblah.com, ,dd invites,subscribed,, master, , , , dd invites,subscribed, ,, , , ff dd
invites, subscribed, , , , , alumni spring 2012 deck: https: www.dropbox.com s,
i69rpofhfsp9t7c practice 20ignition - 20june
.2134.pdf 2109
klkjsdf"
Here is the text, after stipping with 'strip':
"portfolio, derp, hello-world, hello-, -world, founders, mentors, :, ?, %, ,>, , ffib, biff, 1, 12.18.02, 12, 2013, 9874890288, .., ..., ...., , ff, series a, exit, general mailing, fr, , , ,, co founder, pitch_at_palace, ba, _slkdjfl_bf, sdf_jlk, )_(, jim.somedude@blahblah.com, ,dd invites,subscribed,, master, , , , dd invites,subscribed, ,, , , ff dd
invites, subscribed, , , , , alumni spring 2012 deck: https: www.dropbox.com s,
i69rpofhfsp9t7c practice 20ignition - 20june
.2134.pdf 2109
klkjsdf"
Are 'text' and 'clean_text' unchanged? 'True'
So strip removes one whitespace from at a time. So in the OPs case, strip()
is fine. but if things get any more complex, regex and a similar pattern may be of some value for more general settings.
see it in action