• 0

Search engines don't index my dynamic website

Asked by phoe*nix,

 Share

Question

Community Regular

phoe*nix

Posted
Posted (edited)

What is the best way to get search engines, particularly google, to index dynamic content?

I've made a website which uses MySQL and PHP but google can't see the database content. I've read that google is OK with GET requests which is what my site uses. I've also read that a good way to see if a site is easily accessable to search engine robots is to use the lynx browser which i've tried. I can navigate my site fine with lynx but if i put "penally site:www.westwalesholidaycottages.co.uk", without the quotes, into google it won't find anything. Penally is a word which only appears in the database.

I know about mod_rewrite but unfortunately my host doesn't have it enabled :angry: . Has anyone used this? is there an alternative? I was thinking of moving hosts to one which does have it enabled if it was worth it.

Edit: I had an idea about mod_rewrite. the reason my host doesn't have it enabled is because it takes a lot of processing time. Is it possible to add something to the .htaccess file so that apache only translates the address for googlebot and other search bots?

Edit again: ignore my first edit, i realise that wouldn't work :s

Edited by phoe*nix

12 answers to this question

Recommended Posts

  • 0
Enthusiast

teh h4x0r5

Posted
Posted

Yes, mod_rewrite is the solution to this, and yes, you can add something to your .htaccess. However, I am not aware of a way to only rewrite URLs for Googlebot, but, why would it be a problem to rewrite them for users?

Post what you want the link to look like and where you want it to go to, and I will give you the appropriate .htaccess code. ;)

  • 0
Mentor

GatorV

Posted
Posted

I think you are mixed up about what does the Googlebot and other search engines index of your site.

Penally as you said it's only in the database, so in order for google to see it is to implement it in a webpage for example: www.westwalesholidaycottages.co.uk?p=penally and then make a link in your index, and in that page put the word penally, that way google will find it.

If you wan't to make a search for your site your best way is to program yourself one or make all of the content static and suscribe to google so google can search your site (only if is static).

  • 0
Community Regular

phoe*nix

Posted
  • Author
Posted

the point of redirecting only search bots is because mod_rewrite takes a lot of processing power because, i'm guessing here, everytime the server receives a request for say www.westwalesholidaycottages.co.uk/browse/b/1/pl/p it has to convert that to www.westwalesholidaycottages.co.uk/browse.php?b=1&pl=p. So my host has it disabled.

i realised redirecting only search bots wouldn't work.

in theory google should find penally because, on the first page there's a link to the browse page and from there, there is a link to searchresult.php?b=1&pl=p, where pl=p is a list of cottages where their names begin with p. penally would appear in that search result page.

the problem is, my host doesn't have mod_rewrite enabled. is there an alternative to using mod_rewrite?

  • 0
Grand Master

Garry

Posted
Posted
  teh h4x0r5 said:
That's the whole point of mod_rewrite; it makes it so that dynamic content can be indexed.

586772182[/snapback]

I think you'll find that this is *not* the whole point of mod_rewrite... Just one of its many uses.

  • 0
Rookie

andrewkendall

Posted
Posted

I'm a big fan of mod_rewrite but I use it mainly for usability.

Since it's been implemented I've seen a noticeable rise in inbound links, people seem much more happy with linking the tidy URLs:

http://www.andrewkendall.com/pages/photogallery/742/9.php

vs

http://www.andrewkendall.com/pages/photoga...picturenumber=9

That said, I've never found a problem with Google indexing dynamic pages, especially since I started using Google Sitemaps.

i.e. "My Chemical Romance Pictures" shows up fine.

  • 0
Community Regular

phoe*nix

Posted
  • Author
Posted
  Computer Guru said:
sitemaps are the answer. just write a php script that automatically updates hte xml after every change.

586775905[/snapback]

Can you elaborate?

Can you add dynamic URLs to sitemaps? for example: <loc>http://www.westwalesholidaycottages.co.uk/housedetails.php?propref=llan</loc>

  • 0
Mentor

dougal.s

Posted
Posted

I'm in agreement with all the guys saying to *not* use ?foo=foo&foo2=foo2 etc.

Check out this tutorial on how to make clean URIs without mod_rewrite: http://www.evolt.org/article/Making_clean_...d_PHP/18/22880/.

Dougal.

This topic is now closed to further replies.
 Share
Go to question listing

  • Recently Browsing   0 members

    • No registered users viewing this page.

  • Posts

    • Politically funny images of the World

      By +primortal · Posted

    • Microsoft: Edge performs better than Google Chrome on Windows in ad blocking and more

      By hellowalkman · Posted

      Microsoft: Edge performs better than Google Chrome on Windows in ad blocking and more by Sayan Sen Recently, Google put out a performance report for its Chrome web browser explaining how the changes it made to memory management and caching mechanisms managed to make it the fastest it has ever been. Google is not the only one to do it recently, as Microsoft too made similar claims about performance boosts earlier in the year in April. If you were trying to pick between the two, Microsoft is trying to make it easy for you to decide. In a recent blog post titled "Microsoft Edge browser: The fast, smart alternative to Chrome," the tech giant has laid out all the reasons you should pick its browser over Google Chrome. As pointed out in our first paragraph, performance is something these companies often try to brag about, and for good reasons, because people do want to get a snappier experience. This could explain why Mozilla's Firefox has been left behind over time in terms of market share, as it has not been able to keep up with Chromium-based browsers in the performance department. Microsoft says that Edge is better for your Windows PC than Google's Chrome, as it is a "Microsoft product" that "integrates closely with Microsoft Windows," which helps with "performance benefits." Besides that, the Redmond company also points out other "speed and efficiency" features of Edge, like sleeping tabs that helped save 'over 7 trillion megabytes of memory' in 2024. And Edge is also said to "help with smooth performance so that add-ons like Microsoft Translator, ad blockers, and password managers work efficiently." Microsoft adds how the "optimized architecture of Edge helps with lower CPU load and overall resource consumption, providing a smooth experience, particularly on low-spec devices" and also how its "efficient resource allocation helps minimize slowdowns." Aside from performance, productivity is another area where Microsoft says Edge excels. That is thanks to Microsoft 365 integration and other AI tools. In the end of the blog post, Microsoft suggests that this is the right time to switch to Edge from Google Chrome. You can find the post here on Microsoft's official website.
    • The list of Mac devices rumored to get macOS 26

      By Circaflex · Posted

      At some point, hardware becomes obsolete and we cant blame vendors for dropping support. Much of the models dropped, run Intel hardware which means they can install something else on the device if they truly want to keep using it.
    • Microsoft shares "immediate remediation" script if you deleted this Windows system folder

      By ecmarm · Posted

      Well, that was fun: PS C:\WINDOWS\system32> Install-Script -Name Set-InetpubFolderAcl WARNING: Unable to download from URI 'https://go.microsoft.com/fwlin...627338&clcid=0x409' to ''. WARNING: Unable to download the list of available providers. Check your internet connection. PackageManagement\Install-PackageProvider : No match was found for the specified search criteria for the provider 'NuGet'. The package provider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags. At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7468 char:21 + ... $null = PackageManagement\Install-PackageProvider -Name $script:N ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (Microsoft.Power...PackageProvider:InstallPackageProvider) [Install-PackageProvider], Exception + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackageProvider PackageManagement\Import-PackageProvider : No match was found for the specified search criteria and provider name 'NuGet'. Try 'Get-PackageProvider -ListAvailable' to see if the provider exists on the system. At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7474 char:21 + ... $null = PackageManagement\Import-PackageProvider -Name $script:Nu ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidData: (NuGet:String) [Import-PackageProvider], Exception + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.ImportPackageProvider WARNING: Unable to download from URI 'https://go.microsoft.com/fwlin...627338&clcid=0x409' to ''. WARNING: Unable to download the list of available providers. Check your internet connection. PackageManagement\Get-PackageProvider : Unable to find package provider 'NuGet'. It may not be imported yet. Try 'Get-PackageProvider -ListAvailable'. At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7478 char:30 + ... tProvider = PackageManagement\Get-PackageProvider -Name $script:NuGet ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (Microsoft.Power...PackageProvider:GetPackageProvider) [Get-PackageProvider], Exceptio n + FullyQualifiedErrorId : UnknownProviderFromActivatedList,Microsoft.PowerShell.PackageManagement.Cmdlets.GetPackageProvider Install-Script : NuGet provider is required to interact with NuGet-based repositories. Please ensure that '2.8.5.201' or newer version of NuGet provider is installed. At line:1 char:1 + Install-Script -Name Set-InetpubFolderAcl + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: ( [Install-Script], InvalidOperationException + FullyQualifiedErrorId : CouldNotInstallNuGetProvider,Install-Script
    • The list of Mac devices rumored to get macOS 26

      By Skyfrog · Posted

      Because Apple has never discontinued Mac models before now?

  • Recent Achievements

    • Week One Done
      abortretryfail earned a badge
      Week One Done
    • First Post
      Mr bot earned a badge
      First Post
    • First Post
      Bkl211 earned a badge
      First Post
    • One Year In
      Mido gaber earned a badge
      One Year In
    • One Year In
      Vladimir Migunov earned a badge
      One Year In

  • Popular Contributors

    1. 1
      +primortal
      488
    2. 2
      +FloatingFatMan
      257
    3. 3
      snowy owl
      247
    4. 4
      ATLien_0
      222
    5. 5
      +Edouard
      191
    Show More

  • Tell a friend

    Love Neowin? Tell a friend!