Scraping data from Magento without privileged access or trust
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
There appear to be a number of ways of scraping product data from a Magento site, but all seem to have their upsides and downsides.
We deal with sites who have little to no technical resource, but who have given us permission to scrape their product catalog. There appear to be 3 different ways of doing this, none of which really work:
- Manual web scraping - developer intensive, requires updating when the theme changes.
- Magento Web API - requires setting up an API user, too technical for many users.
- Magento Plugin - too technical for many users, exposes sensitive business data so many companies won't do this.
Are we missing something? Is there a better alternative, or are there ways of changing any of the above 3 to be better for scraping?
For example, is it possible to provide a link to a 'one-click-setup' like process for API access? Shopify do this in a nice way using OAuth and permission scopes, so we can give our partners a link that will give us read only access to just their product catalog, in a way that non-technical users can use.
api extensions
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
bumped to the homepage by Community♦ 16 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
There appear to be a number of ways of scraping product data from a Magento site, but all seem to have their upsides and downsides.
We deal with sites who have little to no technical resource, but who have given us permission to scrape their product catalog. There appear to be 3 different ways of doing this, none of which really work:
- Manual web scraping - developer intensive, requires updating when the theme changes.
- Magento Web API - requires setting up an API user, too technical for many users.
- Magento Plugin - too technical for many users, exposes sensitive business data so many companies won't do this.
Are we missing something? Is there a better alternative, or are there ways of changing any of the above 3 to be better for scraping?
For example, is it possible to provide a link to a 'one-click-setup' like process for API access? Shopify do this in a nice way using OAuth and permission scopes, so we can give our partners a link that will give us read only access to just their product catalog, in a way that non-technical users can use.
api extensions
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
bumped to the homepage by Community♦ 16 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
There appear to be a number of ways of scraping product data from a Magento site, but all seem to have their upsides and downsides.
We deal with sites who have little to no technical resource, but who have given us permission to scrape their product catalog. There appear to be 3 different ways of doing this, none of which really work:
- Manual web scraping - developer intensive, requires updating when the theme changes.
- Magento Web API - requires setting up an API user, too technical for many users.
- Magento Plugin - too technical for many users, exposes sensitive business data so many companies won't do this.
Are we missing something? Is there a better alternative, or are there ways of changing any of the above 3 to be better for scraping?
For example, is it possible to provide a link to a 'one-click-setup' like process for API access? Shopify do this in a nice way using OAuth and permission scopes, so we can give our partners a link that will give us read only access to just their product catalog, in a way that non-technical users can use.
api extensions
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
There appear to be a number of ways of scraping product data from a Magento site, but all seem to have their upsides and downsides.
We deal with sites who have little to no technical resource, but who have given us permission to scrape their product catalog. There appear to be 3 different ways of doing this, none of which really work:
- Manual web scraping - developer intensive, requires updating when the theme changes.
- Magento Web API - requires setting up an API user, too technical for many users.
- Magento Plugin - too technical for many users, exposes sensitive business data so many companies won't do this.
Are we missing something? Is there a better alternative, or are there ways of changing any of the above 3 to be better for scraping?
For example, is it possible to provide a link to a 'one-click-setup' like process for API access? Shopify do this in a nice way using OAuth and permission scopes, so we can give our partners a link that will give us read only access to just their product catalog, in a way that non-technical users can use.
api extensions
api extensions
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
asked Aug 17 '15 at 17:20
danpalmerdanpalmer
1213
1213
bumped to the homepage by Community♦ 16 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 16 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Not sure why magento plugin in and of itself would be too technical, especially if instructed to install via magento connect.
Which could build an accessible XML feed for you so you could scrape/retrieve the feed via HTTP without worrying about a changing theme layer.
I don't think this is the one click answer you're looking for, but an 'alternative' solution could be to have clients upload a custom script that you provide.
That script could be run via cron, and would perform periodic dumps of specified DB tables (i.e. no tables which contain 'sensitive business data').
Each dump could be retrieved via ssh/sftp if you have access to that, a public facing folder / email if not. Setting up a crontask via cpanel would be pretty easy for the average user.
That would give you the most complete dataset, although not without its glaring downsides.
As a sidenote, xpath parser for webscraping is an elegant tool, and could be implemented in a way to be mostly theme agnostic if it comes to that.
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "479"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f78946%2fscraping-data-from-magento-without-privileged-access-or-trust%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Not sure why magento plugin in and of itself would be too technical, especially if instructed to install via magento connect.
Which could build an accessible XML feed for you so you could scrape/retrieve the feed via HTTP without worrying about a changing theme layer.
I don't think this is the one click answer you're looking for, but an 'alternative' solution could be to have clients upload a custom script that you provide.
That script could be run via cron, and would perform periodic dumps of specified DB tables (i.e. no tables which contain 'sensitive business data').
Each dump could be retrieved via ssh/sftp if you have access to that, a public facing folder / email if not. Setting up a crontask via cpanel would be pretty easy for the average user.
That would give you the most complete dataset, although not without its glaring downsides.
As a sidenote, xpath parser for webscraping is an elegant tool, and could be implemented in a way to be mostly theme agnostic if it comes to that.
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
add a comment |
Not sure why magento plugin in and of itself would be too technical, especially if instructed to install via magento connect.
Which could build an accessible XML feed for you so you could scrape/retrieve the feed via HTTP without worrying about a changing theme layer.
I don't think this is the one click answer you're looking for, but an 'alternative' solution could be to have clients upload a custom script that you provide.
That script could be run via cron, and would perform periodic dumps of specified DB tables (i.e. no tables which contain 'sensitive business data').
Each dump could be retrieved via ssh/sftp if you have access to that, a public facing folder / email if not. Setting up a crontask via cpanel would be pretty easy for the average user.
That would give you the most complete dataset, although not without its glaring downsides.
As a sidenote, xpath parser for webscraping is an elegant tool, and could be implemented in a way to be mostly theme agnostic if it comes to that.
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
add a comment |
Not sure why magento plugin in and of itself would be too technical, especially if instructed to install via magento connect.
Which could build an accessible XML feed for you so you could scrape/retrieve the feed via HTTP without worrying about a changing theme layer.
I don't think this is the one click answer you're looking for, but an 'alternative' solution could be to have clients upload a custom script that you provide.
That script could be run via cron, and would perform periodic dumps of specified DB tables (i.e. no tables which contain 'sensitive business data').
Each dump could be retrieved via ssh/sftp if you have access to that, a public facing folder / email if not. Setting up a crontask via cpanel would be pretty easy for the average user.
That would give you the most complete dataset, although not without its glaring downsides.
As a sidenote, xpath parser for webscraping is an elegant tool, and could be implemented in a way to be mostly theme agnostic if it comes to that.
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
Not sure why magento plugin in and of itself would be too technical, especially if instructed to install via magento connect.
Which could build an accessible XML feed for you so you could scrape/retrieve the feed via HTTP without worrying about a changing theme layer.
I don't think this is the one click answer you're looking for, but an 'alternative' solution could be to have clients upload a custom script that you provide.
That script could be run via cron, and would perform periodic dumps of specified DB tables (i.e. no tables which contain 'sensitive business data').
Each dump could be retrieved via ssh/sftp if you have access to that, a public facing folder / email if not. Setting up a crontask via cpanel would be pretty easy for the average user.
That would give you the most complete dataset, although not without its glaring downsides.
As a sidenote, xpath parser for webscraping is an elegant tool, and could be implemented in a way to be mostly theme agnostic if it comes to that.
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
edited Nov 21 '17 at 13:07
Teja Bhagavan Kollepara
2,98241949
2,98241949
answered Aug 17 '15 at 18:20
bakubaku
1229
answered Aug 17 '15 at 18:20
bakubaku
1229
answered Aug 17 '15 at 18:20
bakubaku
1229
1229
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
add a comment |
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
Thanks for your reply! Unfortunately Magento Connect looks too complicated for some of our partners, they often use contractors to set up Magento, and aren't able to do things like this. Also it doesn't solve the permissions issue, that plugins can read anything they want. We already use XPath, sitemaps, and lots of other ways to scrape data from the pages, but Magento themes differ enough on the sites we already do this for that we can't share much if any scraping code between them.
– danpalmer
Aug 18 '15 at 8:37
add a comment |
Thanks for contributing an answer to Magento Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f78946%2fscraping-data-from-magento-without-privileged-access-or-trust%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
