Christian Heilmann

A code snippet to scrape all headings and their target URLs from a markdown generated page

Friday, February 19th, 2021 at 7:54 pm

When you use Markdown to write your documentation most static page generators will generate IDs for each of the headings in the document to allow you to navigate directly to them.

Markdown: 
## Gerbils and other rodents
HTML: 
<h2 id="gerbils-and-other-rodents">Gerbils and other Rodents</h2>

To go directly there you can then use https://example.com#gerbils-and-other-rodents if you published at example.com.

The other day I was asked to create a list of all the links in the What’s new in Devtools 89 document, which is generated from Markdown. The list should be the text of the headline followed by the full URL to get to that part of the document. This was to batch generate some shortURLs from them.

I am pretty sure there are a lot of clever ways to do that by scraping but as I like my browser environment, I just used the Console to do that. Here’s the script that you can paste into the Console:

let out = '';
$$(':is(h1,h2,h3,h4,h5,h6)[id]').forEach(elm => {
   out += `${elm.innerText}
${document.location.href}#${elm.id}
` 
});
copy(out);

You can see it in action in the following GIF:

Running the script in the console to get all the heading information

The next step was to store this as a Snippet and next time I just need to run that.

Share on Mastodon (needs instance)

Share on Twitter

Newsletter

Check out the Dev Digest Newsletter I write every week for WeAreDevelopers. Latest issues:

160: Graphs and RAGs explained and VS Code extension hacks Graphs and RAG explained, how AI is reshaping UI and work, how to efficiently use Cursor, VS Code extensions security issues.
159: AI pipelines, 10x faster TypeScript, How to interview How to use LLMs to help you write code and how much electricity does that use? Is your API secure? 10x faster TypeScript thanks to Go!
158: 🕹️ Super Mario AI 🔑 API keys in LLMs 🤙🏾 Vibe Coding Why is AI playing Super Mario? How is hallucinating the least of our worries and what are rules for developing Safety Critical Code?
157: CUDA in Python, Gemini Code Assist and back-dooring LLMs We met with a CUDA expert from NVIDIA about the future of hardware, we look at how AI fails and how to play pong on 140 browser tabs.
156: Enterprise dead, all about Bluesky and React moves on! Learn about Bluesky as a platform, how to build a React App and how to speed up SQL. And play an impossible game in the browser.

My other work: