Sigmera for Data Analysts
Most of an analyst's time goes to the file, not the analysis. A raw export arrives with duplicate rows, three spellings of the same category, and a phone column in five formats, and none of it groups or joins cleanly until it is fixed. Sigmera handles the quick, one-off cleanup in a browser tab — dedupe, standardise values, split or join columns, convert between CSV and Excel — and the dataset never leaves your device, which matters when it holds personal data.
The data problems you hit
- The prep tax is real and measured. Anaconda's 2020 State of Data Science survey found respondents spend on average 45% of their time getting data ready — loading and cleansing — before they can use it to build models and visualisations. A one-off CSV that does not warrant a full pipeline still eats an afternoon of manual fixes.
- Inconsistent categorical values silently break every group-by. A Region column holding "Paris", "paris", and "Paris " with a trailing space is three groups to a pivot, not one, so counts split, joins drop rows, and a total that should be clean scatters across near-identical labels. The values look the same on screen and the aggregate is wrong anyway.
- Duplicate rows skew every aggregate they touch. A dataset stitched together from several exports carries the same record more than once, and a naive count, sum, or average inherits the inflation. Deduplicating on a stable key before you aggregate is the difference between a number you can defend and one you cannot.
- The privacy shortcut is a trap. The fast way to clean a messy file is to paste it into whatever free online cleaner ranks first — but that makes the vendor a processor of whatever the file contains. GDPR Article 28(1) requires a controller to use only processors that provide sufficient guarantees of appropriate technical and organisational measures, a bar a random web form has not cleared, and a dataset of real people is exactly the kind you should not hand over casually.
- CSV-to-Excel round trips corrupt the data quietly. Opening a CSV in Excel to tidy it can drop the leading zeros off postal codes and IDs, reformat long numbers into scientific notation, and reinterpret dates by locale — so the file you analyse no longer matches the file you were given, and the damage is invisible until a join fails.
What you can do in Sigmera
Before any count, sum, or average, collapse the repeats a merged export carries. Matching on a stable key — an ID or a normalised email — and dropping case-insensitive duplicates gives you an aggregate that reflects the real records rather than how many times each one was exported.
Fix the spelling and whitespace variants that split a group-by. Find and replace across the whole column folds "N/A", "na", and blank into one value, or rewrites "USA" and "United States" to a single label, so the pivot counts groups the way you mean them.
Split a combined "Last, First" field into two columns for a join key, or stitch City and Country into one label for a chart axis. Doing it in a dedicated tool handles the compound cases a LEFT/RIGHT formula mangles, and you see every row before you export.
Move a file between formats without the silent corruption Excel introduces on the way in. Converting in the browser writes UTF-8 and preserves the text as-is, so leading zeros, long IDs, and dates survive the trip intact instead of being reinterpreted.
Before you load a file into a notebook or BI tool, plot a column or two to catch the obvious problems — a spike from duplicates, a category that should not exist, a date range that is wrong. A quick in-browser chart flags the data issues worth fixing first.
When you need to join two datasets on a phone number or reference, standardise the format first. Rewriting a phone column to E.164 — one consistent shape — makes the join actually match the rows that belong together instead of missing the same value written two ways.
Sources
Frequently asked questions
- Is Sigmera a replacement for OpenRefine or pandas?
- No, and it is honest to say where it is not. For a reproducible pipeline, a transformation you will re-run every week, or a file too large to hold in memory, a scripted tool like pandas or a dedicated app like OpenRefine is the right choice. Sigmera is for the one-off — the export that needs deduping, a column standardised, or a format fixed before you analyse it — done in a browser tab without setting up an environment or uploading the file. Many analysts use both: the script for the pipeline, the browser tool for the quick fix.
- How large a file can Sigmera handle?
- There is no row or file-size cap built into the tools — the ceiling is your own machine's memory, because nothing is queued against an upload limit or sent to a server to process. A few hundred thousand rows is comfortable on a normal laptop. Beyond what fits in browser memory, a scripted or database approach is the better fit.
- Is it GDPR-safe to clean a dataset with personal data this way?
- Cleaning a dataset is processing under GDPR wherever you do it, and you stay the controller with your lawful basis and retention duties unchanged. What browser-side cleaning changes is that no other party receives the file. Sigmera reads and rewrites it with JavaScript on your own machine, so no vendor becomes a processor. Sending the same file to a cloud cleaner would create that processor relationship, and Article 28(1) requires a controller to use only processors offering sufficient guarantees of appropriate technical and organisational measures.
- Why clean a CSV before visualising it?
- Because dirty data produces confident, wrong charts. Duplicate rows inflate a bar, inconsistent category labels split one group into several, and a stray value stretches an axis so the real distribution is unreadable. Deduping and standardising the columns you are about to plot takes a minute and stops you drawing a conclusion from an artefact of the file rather than the data.
- Will converting a file change my data types or drop leading zeros?
- That is exactly the corruption the browser converter avoids. Opening a CSV directly in Excel is where leading zeros vanish, long numbers turn into scientific notation, and dates get reinterpreted by locale. Sigmera treats the values as text and writes UTF-8, so an ID like 00734 or a date stays as it was written. When you need a workbook, convert with the tool rather than by opening the raw CSV in Excel.
- Do I need an account to use the tools?
- No account is needed to preview a cleanup, and previewing is free and unlimited — you can load a file, dedupe it, standardise columns, and see the result without signing up. A free account covers three downloads a month when you want to export the finished file, with paid plans above that. Only the export counts against the plan; the file itself is never uploaded.