url散列算法原理

by Ty Irvine

由Ty Irvine

如何列出与网站相关的所有URL (How to List Out All URLs Associated With a Website Fast-ish)

So you need a list containing all the URLs for a website? Are you doing some redirects perhaps? Hit the limit on ? Cool, me too. I’ve got just the tool for you that’ll get it done at about the same speed as XML Sitemaps, but you’ll look way cooler doing it.

因此，您需要一个包含网站所有URL的列表吗？ 您是否正在执行一些重定向？ 达到的极限？ 酷，我也是。 我只是为您提供了可以以与XML Sitemaps相同的速度完成它的工具，但是您看起来会更酷。

教程实际开始的位置 (Where the tutorial actually starts)

To get your list of URLs, we’re going to use Wget!

为了获得您的URL列表，我们将使用Wget！

什么是Frigg Wget？ (What the Frigg is Wget?)

“Wget is a free software package for retrieving files using HTTP, HTTPS, and FTP, the most widely-used Internet protocols.” —

“ Wget是一个免费软件包，可以使用HTTP，HTTPS和FTP(使用最广泛的Internet协议)来检索文件。” —

And you can also use it to request a big list of URLs associated with a domain.‍

您还可以使用它来请求与域相关联的URL的大列表。

‍1。 安装Wget (‍1. Installing Wget)

To install Wget if you haven’t already, you’re going to need first to install ; aka Brew. ? Brew is a package manager, meaning it installs software for you and manages it. You can check out the instructions on their website or just follow the ones below.

如果尚未安装Wget，则需要先安装 。 aka Brew。 ？ Brew是一个程序包管理器，这意味着它会为您安装和管理软件。 您可以在其网站上查看说明，也可以按照以下说明进行操作。

安装Brew (Install Brew)

Paste this into a Terminal Prompt and hit enter twice ⮐ (It may ask you for a password.)

将其粘贴到终端提示中，然后按两次Enter键(可能会要求您输入密码)。

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"‍

安装Wget (Install Wget)

Now that you have Brew installed it’s time to install Wget. Paste this into a Terminal Prompt and hit enter ⮐

现在已经安装了Brew，现在该安装Wget了。 将其粘贴到终端提示中，然后按Enter键

brew install wget

2.时间获取“ Dem URL” (2. Time To Get ‘Dem URLs‍)

Now with Wget installed we simply download the website and then display all of its URLs. Start by downloading the website you’d like with

现在安装了Wget之后，我们只需下载网站，然后显示其所有URL。 首先下载您想要的网站

Wget -r www.shutterandcode.com

Then once the download is complete we’ll list out the URLs with

然后，一旦下载完成，我们将列出网址

Find www.shutterandcode.com

(Make sure to use the same website domain as what was downloaded).

(确保使用与下载的网站相同的网站域)。

结论 (Conclusion)

After a series of casual tests pitting Wget against XML Sitemaps using smaller websites, I found that they are both pretty much on par with each other. Occasionally one would be faster than the other but overall they both had similar speeds.‍

在使用较小的网站对Wget与XML Sitemaps进行一系列偶然测试之后，我发现它们几乎可以相互媲美。 有时候，一个速度会比另一个速度快，但是总体而言，它们的速度都差不多。

If you’d like to know more about Wget commands simply type this into your prompt

如果您想了解有关Wget命令的更多信息，只需在提示符下键入

wget --help

I hope you enjoyed reading this! Don’t forget to like, comment, and subscribe! ?

希望您喜欢阅读！ 不要忘记喜欢，评论和订阅！ ？

p.s. don’t actually feel obligated to like, comment, and or subscribe because it is simply a joke for YouTubers :)

ps实际上并没有义务去喜欢，评论或订阅，因为这只是YouTubers的笑话：)

UPDATE: if you don’t want the site actually to download to your computer add in ‘ — spider’ after ‘wget’ like

更新：如果您不希望站点实际下载到计算机上，则在“ wget”之后添加“ — Spider”

wget -r --spider www.example.com

Check out the original post and the rest of the Snippets! series at

查看原始帖子和其他片段！ 系列在

翻译自:

url散列算法原理