Custom content

Hanjiewu · June 21, 2024, 6:16am

I am excited to run multiqc to generate a custom content report. I installed multiqc (v1.22.2) and it has no problem to generate the bakta report. However, as I ventured to custom content, I downloaded all 3 example files from [test-data/data/custom_content/with_config/issue_1205 at main · MultiQC/test-data · GitHub](https://test data) and executed:

multiqc -c multiqc_config.yaml .

but I got the following message:

/// MultiQC v1.22.2
            config | Loading config settings from: multiqc_config.yaml
            config | Loading config settings from: multiqc_config.yaml
       file_search | Only using modules: custom_content
       file_search | Search path: /mnt/nas96T_10G/hanjie/playground/multiqc
        searching | ████████████████████████████████████████ 100% 3/3
              plot | custom_content/custom_content.py: Invalid Table plot configuration {'id': 'summary_assembly_metrics_plot', 'table_title': 'De novo assembly metrics', 'namespace': 'De novo assembly metrics', 'format': '{:.0f}', 'col1_header': 'Sample', 'title': 'De novo assembly metrics'}:
              plot | • unrecognized field 'format'. Available fields: id, table_title, title, height, width, square, logswitch, logswitch_active, logswitch_label, cpswitch, cpswitch_c_active, cpswitch_counts_label, cpswitch_percent_label, xLog, yLog, xlog, ylog, data_labels, xTitle, yTitle, xlab, ylab, xsuffix, ysuffix, tt_suffix, xLabFormat, yLabFormat, yLabelFormat, xlab_format, ylab_format, tt_label, xDecimals, yDecimals, decimalPlaces, x_decimals, y_decimals, tt_decimals, xmin, xmax, ymin, ymax, xFloor, xCeiling, yFloor, yCeiling, x_clipmin, x_clipmax, y_clipmin, y_clipmax, save_data_file, namespace, save_file, raw_data_fn, defaultsort, sort_rows, only_defined_headers, col1_header, no_violin, scale, min

I need help, please teach me how to create a yaml config that will render my own tsv files.

vlad.savelyev · June 21, 2024, 7:12pm

Dang, I should have updated the examples Thanks for pointing this out this issue! Fixed them now with Fix table custom content example · MultiQC/test-data@11af301 · GitHub

Hanjiewu · June 22, 2024, 1:07am

It’s not working for me with the updated yaml. I downloaded all 3 files and ran

multicq -c .

this time I got

    /// MultiQC v1.22.2

    config | Error parsing config YAML: mapping values are not allowed here in "multiqc_config.yaml", line 212, column 75 
Traceback (most recent call last):
  File "/root/miniconda3/bin/multiqc", line 8, in <module>
    sys.exit(run_multiqc())
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/__main__.py", line 21, in run_multiqc
    multiqc.run_cli(prog_name="multiqc")
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/multiqc.py", line 448, in run_cli
    result = run(*analysis_dir, clean_up=clean_up, cfg=cfg)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/multiqc.py", line 493, in run
    update_config(*analysis_dir, cfg=cfg)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/core/update_config.py", line 93, in update_config
    config.load_user_files()
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/config.py", line 230, in load_user_files
    load_config_file("multiqc_config.yaml")
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/config.py", line 305, in load_config_file
    new_config: Optional[Dict] = pyaml_env.parse_config(str(path))
  File "/root/miniconda3/lib/python3.10/site-packages/pyaml_env/parse_config.py", line 110, in parse_config
    return yaml.load(conf_data, Loader=loader)
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/composer.py", line 58, in compose_document
    self.get_event()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/parser.py", line 118, in get_event
    self.current_event = self.state()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/parser.py", line 193, in parse_document_end
    token = self.peek_token()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 129, in peek_token
    self.fetch_more_tokens()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "multiqc_config.yaml", line 212, column 75

I am confused that all headers, except for the first column, and the file name to be processed are listed in this example, which by the way also encoutered the same issue/error message, but in your updated yaml, no headers are listed except for the “% Mapped reads (viral)”. What is the proper way for multiqc to process a tsv file and plot it in table?

I am uploading the tsv file and custom config here
barcode01_homopolished_amr_ncbi.tsv (2.2 KB)
multiqc_config.yaml (1.5 KB)
when I ran multiqc I got noting and this this the message I got:

/// MultiQC v1.22.2

            config | Loading config settings from: multiqc_config.yaml
            config | Loading config settings from: multiqc_config.yaml
       file_search | Search path: /tmp/multiqc
        searching | ████████████████████████████████████████ 100% 2/2
      exec_modules | No analysis results found. Cleaning up…
      exec_modules | MultiQC complete

Please help me to figure out how to create a multiqc report using my own data.

vlad.savelyev · June 22, 2024, 12:15pm

@Hanjiewu,

In your TSV file, please remove the comment character from the table header, i.e. repalce

#FILE	SEQUENCE	START	END	STRAND	GENE	COVERAGE	COVERAGE_MAP	GAPS	%COVERAGE	%IDENTITY	DATABASE	ACCESSION	PRODUCT	RESISTANCE

With

FILE	SEQUENCE	START	END	STRAND	GENE	COVERAGE	COVERAGE_MAP	GAPS	%COVERAGE	%IDENTITY	DATABASE	ACCESSION	PRODUCT	RESISTANCE

The reason for that is that we also allow provide an inline plot config fenced with #, e.g. it could have been:

#id: 'amr_results'
#section_name: 'AMR Results'
#description: "Antimicrobial resistance gene predictions from sequencing data"
#plot_type: 'table'
#pconfig:
#    id: 'amr_results'
#    namespace: 'AMR Results'
FILE	SEQUENCE	START	END	STRAND	GENE	COVERAGE	COVERAGE_MAP	GAPS	%COVERAGE	%IDENTITY	DATABASE	ACCESSION	PRODUCT	RESISTANCE

That being said, we should show a more clear warnings about that (added with Show error message if failed to parse custom content header · MultiQC/MultiQC@d736846 · GitHub).

We probably should also be able to handle to guess if the line behind # contains table columns, and handle your case without errors. I made this update in the code in Custom content: allow hash-fenced table columns by vladsavelyev · Pull Request #2649 · MultiQC/MultiQC · GitHub

But for now, please replace #FILE with FILE.

Let me know if it works!

vlad.savelyev · June 22, 2024, 12:18pm

One more thing. The first column of the table should be SAMPLE (or any other unique row identifier), and contain unique values for each row. In your case, all values are identical and equal to barcode01_homopolished.fasta, so they all collapse in the report into one row.

Hanjiewu · June 24, 2024, 1:23am

@vlad.savelyev , unfortunatelyit it still showed “No analysis results found. Cleaning up…” even after removing the # in the header. Then I noticed there was an update to MultiQC, so I upgraded it to v1.22.3, and it worked. I guess there was a bug handling custom contents in v1.22.2. Thank you for the guidance and the update.

I am still confused about when to add headers and file names to be processed in the config, as shown in the example here, and when to omit the headers and file names to be processed, like this example which by the way still not working even with the updated multiqc. Here are the error message:


/// MultiQC v1.22.3

            config | Error parsing config YAML: mapping values are not allowed here
  in "multiqc_config.yaml", line 212, column 75
Traceback (most recent call last):
  File "/root/miniconda3/bin/multiqc", line 8, in <module>
    sys.exit(run_multiqc())
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/__main__.py", line 21, in run_multiqc
    multiqc.run_cli(prog_name="multiqc")
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/multiqc.py", line 448, in run_cli
    result = run(*analysis_dir, clean_up=clean_up, cfg=cfg)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/multiqc.py", line 493, in run
    update_config(*analysis_dir, cfg=cfg)
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/core/update_config.py", line 93, in update_config
    config.load_user_files()
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/config.py", line 232, in load_user_files
    load_config_file("multiqc_config.yaml")
  File "/root/miniconda3/lib/python3.10/site-packages/multiqc/config.py", line 315, in load_config_file
    new_config: Optional[Dict] = pyaml_env.parse_config(str(path))
  File "/root/miniconda3/lib/python3.10/site-packages/pyaml_env/parse_config.py", line 110, in parse_config
    return yaml.load(conf_data, Loader=loader)
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/composer.py", line 58, in compose_document
    self.get_event()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/parser.py", line 118, in get_event
    self.current_event = self.state()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/parser.py", line 193, in parse_document_end
    token = self.peek_token()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 129, in peek_token
    self.fetch_more_tokens()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/root/miniconda3/lib/python3.10/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "multiqc_config.yaml", line 212, column 75

vlad.savelyev · June 24, 2024, 11:27am

Sorry @Hanjiewu, you are right, there is still an issue with that test example Should be fixed now: Fix custom content submodules with config by vladsavelyev · Pull Request #2654 · MultiQC/MultiQC · GitHub

Topic		Replies	Views
How to get several "custom data" images into the same "custom content" section of the report Ask for help multiqc	3	220	December 18, 2023
Custom_plot_config and custom_table_header_config not working Ask for help multiqc	2	24	November 11, 2024
Customising order of modules Ask for help multiqc	3	49	July 20, 2024
Better practices using MultiQC in Quarto Notebooks for sets of images Ask for help multiqc	7	55	September 25, 2024
Can I create histograms with custom_data? Ask for help multiqc	1	174	January 20, 2024

Custom content

Related topics