September 1st, 2009

Oracle delivers native support for Thomson Reuters' OpenCalais service

Posted by Paul Miller @ 5:15 am

Categories: Commercialisation, Semantic Web, Semantic Web Companies

Tags: Thomson Reuters Corp., Oracle Corp., Paul Miller

Thomson Reuters and Oracle today announced support for the media giant’s OpenCalais metadata generation service within release 2 of Oracle Spatial 11g. The integration gives Oracle users and developers direct access to OpenCalais’ natural language processing (NLP) capabilities.

More importantly, perhaps, direct integration with an Enterprise product such as Oracle’s database says much about how far the semantic technology community has come in being able to offer solutions capable of scaling - robustly - to meet Enterprise-scale demands.

Xavier Lopez, Oracle’s Director for Spatial and Semantic Technologies, is quoted in Thomson Reuters’ press release;

“This interoperability lets users quickly process documents in different formats (such as Microsoft Word and Adobe PDF), to extract semantic metadata that can be used for more semantically complete searches in Oracle11g.”

July 14th, 2009

New open source Semantic Web store from Garlik capable of enterprise scale

Posted by Paul Miller @ 5:20 am

Categories: Podcasts, Semantic Web, Semantic Web Companies, Talking Semantics

Tags: Open Source, Semantic Web, RDF, XML, Internet, Software/Web Development, Web Development, Paul Miller

An oft-repeated concern in discussing large-scale deployment of Semantic Web ideas is that of ’scale.’ With many of the better known data stores upon which the Semantic Web depends capable of storing only tens or at best a few hundreds of millions of RDF triples, it can be difficult to argue that the technology is fit for real-world deployment at scale.

There are, of course, different ways of managing data, and it’s not always necessary to store everything in one massive store… but for those concerned about scale today’s news from UK-based Garlik may well put their minds at rest.

The company has taken their internally developed (and massively scalable) RDF triple store and released it to the world under an Open Source license as 4store.

I spoke with the company’s CEO and Head of Architecture just ahead of the launch, to learn more about the system and their motivation behind sharing it.

The result has just been released as a podcast.

June 17th, 2009

Nova Spivack interviews Wolfram Alpha's Russell Foltz-Smith

Posted by Paul Miller @ 1:26 pm

Categories: Commercialisation, Semantic Web, Semantic Web Companies, Semantic Web People

Tags:

Radar Networks attracted a fair degree of attention with their roll-out of Twine, and the company’s CEO has built a reputation as one of the more thoughtful thinkers in the space. Nova took to the stage at the Semantic Technology Conference today, not to talk about his own company or ideas, but to lead a conversation with Russell Foltz-Smith from Wolfram Research.

Wolfram Research, of course, is the company behind the recently launched Wolfram Alpha; a ‘computational knowledge engine’ that attracted a wave of attention that reached into the mainstream media.

“Putting all of the world’s computable knowledge; it sounds impossible… or over-confident, maybe. What is computable knowledge?”

“It’s ’systematic knowledge.’ It can be compared, contrasted, correlated, computed on. It’s not a movie review. Examples are classical physics, financial data and models, weather data and models… It’s not the latest opinion on who Britney Spears is dating. We don’t have a model to do anything with that in our system.”

Nova asks if it’s the difference between objective and subjective… Alpha deals with objective information. ‘Facts,’ almost?

Nova asks about sources, pointing to the example of Tibet; is it a ‘fact’ that Tibet that is part of China, or not… ?

“In the case of geo-political things, and religious things, we have to make choices… and allow the community to let us know whether they agree or not…” Couldn’t the system represent multiple views, tied to the diverse sources? Could we not show the different opinions, and allow the user to make informed decisions themselves?

Nova; “is the world’s computable knowledge infinite?”

Russell; “the foundation of computable knowledge is likely to be finite… The amount of knowledge that can be computed and generated from that is infinite…”

Nova; “I can see that maths could be finite. But geopolitics, health, etc… that’s much, much larger…”

Russell; “The instances seem very complex… Huge, but finite… I don’t want anyone to think we’ll have this done in ten years… It’s a long term thing.”

Nova; “Stephen [Wolfram] reckons it could be done in three years…?”

Nova; “Looking at the back end, the ontology seems to be implicit. I didn’t see any classes, just a lot of instances… a set of facts. As the team grows, how do you prevent people adding facts in different ways?”

Russell; “There are a set of stored facts; things you know about a city. But then there are computed facts that you couldn’t store in a traditional ontology.” Huh?

Nova; “Can you make a statement about what percent of the world’s computable knowledge is there today?”

Russell; “I can’t make a statement…”

Nova; “The syntax is quite interesting… but enigmatic. It wasn’t necessarily that the knowledge wasn’t there, but that I’d asked for it in the wrong way. Can’t you make a manual? … Stephen [Wolfram] said it would be an impossible task to write the manual… or to make a generic natural language on top.”

Nova; “In some cases a naive query will get you the answer, but maybe there’s a need for a layer that helps you when you don’t get what you want…”

Russell; “I think we’re getting close… we’re going to put an API out in the next few weeks, and hopefully someone will build the application using that to parse natural language and translate it for Alpha… Do we spend our time doing that, or putting more data and more models into the system… I reckon our time is best spent adding more data…”

Nova; “Is there a set of schemas or ontologies to link all of this stuff together?”

Russell; “There isn’t an ontology over the whole system… but within a domain there is structure… Is there some grand scheme that we have internally? Not really. The company has been doing this stuff for 23 years, so there’s a bit of a shared understanding internally.”

Examples keep coming back to mathematics… To succeed, Alpha has to offer compelling examples that are far broader…

Nova; “What about reasoning. You’ve said that you can derive additional knowledge. What kinds of reasoning is the system capable of?”

Russell; “I’d call it very simple reasoning. For example changing the currency based upon your geo-location… Is there any weak or strong AI in here? Not really. Could you build something like that? Probably. Will we? I don’t know…”

Nova; “Alpha seems to be a subset of Mathematica capabilities… Would you expand that, and bring a full Mathematica to the Web”

Russell; “It is, and there are plans to extend the capabilities. I don’t know if we’d go to a full-blown Mathematica on the web.”

Russell’s mentioning a subscription service for people working with more data, or needing more compute time. The public web site tends to time out a query in 4-8 (or 48?) seconds… The professional subscription version will have a monthly subscription version that will allow you to compute bigger questions. There will also be a pay-per-use API… and ‘primitive’ advertising. More advanced advertising, based on transactions, to be launched soon.

Nova; “Alpha’s really cool, but I want to do this on my own knowledge… inside an enterprise, inside a government agency…”

Russell; “We can roll out a custom Wolfram Alpha for those who want it behind the firewall. We will also let people upload their own data sets. We need to find a sensible way to let people do this…”

Nova; “There was a lot of hype - possibly my fault - around Alpha being a Google Killer. Obviously it’s not that. It’s something quite different. Who is the user, and what are they using it for?”

Russell; “Use will evolve, and it already has. There’s an obvious use by students, but the school year has just ended.

Nova; “Wolfram Alpha; now even Ph.D’s can cheat on their homework.”

Nova; “Are consumers using it? Obviously they’re having a play, but are they coming back and using it?”

Russell moves off to talk about academic use… Dodging the question?

Nova; “Are the financial capabilities in Alpha differentiated from the capabilities banks and investors already have in their vertical?”

Russell; “more sophisticated than a general finance web site, but probably less sophisticated than you’d find on a terminal in a bank.”

Nova; “Do I really need to know how long it would take an ant to get from San Francisco to Cairo?”

Russell; “Because of the way the system is engineered, it just keeps computing until it runs out of time. With simple queries you’ll get a lot of data. It just keeps computing.”

Nova; “What’s the big challenge, moving forward?”

Russell; “Setting priorities.”

Nova; “So let’s talk about Google. They made some aggressive marketing moves during the Alpha roll-out, and they’re continuing to roll products out to chip away… Do you think that what you’ve built is defensible, just because it’s hard… or can you defend it in other ways?”

Russell; “There are significant barriers to what we’re doing. Someone else could build this… but would they want to? That’s an open question.”

Nova; “Do you hope to work with other companies? Perhaps revenue share with them?”

Russell; “Obviously.”

Nova; “There’s been a lot of interest in how Alpha might connect with open standards and the Semantic Web…”

Russell; “If you want the platform to be used, we’ll have to do some of this stuff… RDF, OWL, etc could play a huge role.”

Nova; “Timeframe?”

Russell; “It’ll depend on pick-up of the API… which is due out in a few weeks.”

Nova; “So what’s the implication for education? It makes it possible to do some things without even thinking…”

Russell; “It’ll be a heated debate for a while… Some things are positive, some negative. There’s going to be a reorientation… It has to happen.”

Nova; “The danger is that if you delegate thinking [inside education] to a computation service… you may not actually understand enough to know if the answer that comes back is correct.”

Russell; “That’s a valid concern.”

Q&A

“You rely more on your computational engine than natural language… but you lay a lot of emphasis on the linguistics in your system. So if it’s not NLP what is it?”

Russell; “Domain linguistics, mainly; mathematical language, engineering language, etc… We think about how people describe things and search in these domains… and crawl the web looking for examples of how people use language in these domains.”

“Stephen is focussed on quality of data, which is important to a lot of people here. There aren’t a lot of tools. In addition to making your data store, I wonder if there might be scope to make some of your data curation tools available to the community, to improve the data out there.”

Russell; “Great point. Can we make these tools genuinely useful to people, without creating a support nightmare…”

June 17th, 2009

Semantic Search Round Table at the Semantic Technology Conference

Posted by Paul Miller @ 9:40 am

Categories: Research, Semantic Web, Semantic Web Companies, Web 3.0

Tags:

Wednesday’s opening Keynote here in San Jose sees Guidewire’s Carla Thompson joined on stage by senior representatives from many of the more interesting players in the Semantic Search space; Tomasz Imielinski from Ask, Peter Norvig from Google, Riza Berkan of Hakia, Scott Provost from Microsoft, William Tunstall-Pedoe of the UK’s True Knowledge, and Andrew Tomkins of Yahoo.

Carla asks each panellist to describe the differentiating aspects of their product in ‘one or two sentences;’

Tomasz; “we receive about three times as many questions as other search companies. We want to answer questions the best we can from multiple sources… using structured and unstructured data.”

Scott; “Bing really focusses on understanding the intent behind queries, and organising the page to help people get to their answer much faster.”

Peter; “We focus on being comprehensive, accurate and fast… so we have to keep on innovating in crawling, ranking, systems engineering. One thing that differentiates us… most companies decide whether to focus on marketing or sales. We focus on engineering.”

Riza; “We are a complete semantic search engine, from the bottom up. We don’t even have an index. We’ve optimised the entire process for semantic operations. We focus on credible and dynamic content, and offer users a new perspective.” Instead of popularity, they focus on credibility.

William; “True Knowledge is a platform that does direct question answering. There’s a knowledge base and an inference engine to answer questions we haven’t seen before.” True Knowledge tries to ‘help when it can, and stay quiet when it can’t,’ as can be seen demonstrated in their recently released Firefox plugin.

Andrew: “Yahoo! is very aggressive about semantic annotation… SearchMonkey is about acquiring semantic information and surfacing it in search results on the page.”

Carla mentions Tom Tague’s keynote from yesterday, where he suggested that ’semantic search is an answer to a question no one is asking’… so “why do we need to change search?”

Tomasz responds, suggesting that users don’t necessarily demand new products that subsequently become successful. eg; no one was asking for the iPod before it launched. “When they see it, they will want it.”

Turning to Google and Yahoo!, Carla asks them “why do we need to change search?”

Peter… “as an industry, satisfaction is very high… but that is just because that’s what people know [now]… People don’t like technology… people like solutions. When we deliver it, people will want it.”

Andrew; “Does search need to change? It already is… Today, on any major search engine, if you search for a restaurant, you’ll see structured information about that restaurant; reviews, phone number, etc… This has been accelerating over the last 3-4 years… When we put this information up, and trigger it correctly, we see far higher levels of engagement from our users than anything else.”

Carla; “it may be a stupid question, but it has to be asked; what is semantic search?”

Scott; “it means a lot of different things. At Powerset we focussed on understanding the meaning in web pages, so we could present them, rank them…”

Carla; “Has Powerset’s focus been diluted by the [Microsoft] acquisition?”

Scott; “No.”

Carla asks Riza; “Someone from Hakia that I spoke to last year said you were the only one doing ‘true semantic search.’ Is that true?”

Riza; “No… Semantic Search can enrich search results… Semantic Search can improve precision/disambiguation… Semantic Search can organise results better. In the future, search will move to more conversational systems, and for that you really need semantic technology.”

Carla; “How do you measure the ’semanticity’ of a search engine?”

Tomasz; “That’s my favourite question… We took a sample of ‘equivalent’ queries from the logs, and ran it to evaluate ranking etc; does the search engine give similar answers to questions like ‘Top 10 songs’ and ‘Top Ten songs,’ etc. Should they?”

Andrew; “It’s incredibly hard to understand what a user will like… if you mess with the logo, it changes the perception of the results… if you make tiny changes, it can have a big impact on perception… When it comes to understanding semantic contact in search, we should identify the task the user is trying to solve… and have a metric that’s aligned to that use case… We can break search queries today into different classes; how do we do when a user is trying to book dinner, or a vacation? Semantic Technology should be judged on its impact based on these task metrics rather than any underlying notions of entity resolution, etc… SearchMonkey, for instance, lets users inject structured data into the process… The information can be incorporated in any way… and change how the results are presented. We have about 15,000 people in our development community, changing the way those results are presented every day.”

Tomasz; “I would expect a semantic search engine to deliver equivalent results to queries that would appear similar to a human being; ‘Top 10 songs’ and ‘Top Ten Songs’ should deliver the same answer. Today in most mainstream search engines they don’t.”

Carla; “Search v. Answers. True Knowledge is billed as ‘the Internet Answer Engine;’ is it necessary to move search to an answer-based format, or has Google trained users to think in keywords?”

William; “We support both keyword search and full-text questions. It’s important to answer users’ questions.”

Peter; “Different types of answers are appropriate for different types of questions; sometimes the answer is a fact, or a page, or a series of results to support a process of study. To say there’s going to be one technology or one type of answer doesn’t make sense.”

Riza; “You could be asking a ‘where,’ ‘why,’ ‘how’ type of question. Questions are important, and the search engine needs to be able to interpret the mode of the question and return results appropriately.”

Carla; “You mentioned talking about the credibility of search results. How do you define a ‘credible’ search result, and how much of a need is there really? I’m not hearing users question the credibility of search results they see today.”

Riza; “Practically, credibility is important in ’serious’ subjects; medical information, etc. You want to know where the results come from and how credible they are. When it comes to credible content, you can’t really do a statistical search or have a ‘popularity vote,’ because much credible content isn’t ‘popular.’

Scott; “People’s expectations for credibility are different depending upon the query. If you ask an ‘instant answer’ type query you expect the answer to be credible. If you do a broader search, you expect a mix of results to be returned”

William; “If a system understands structured knowledge, it can understand when different sources contradict one another”

Riza; “A system doesn’t need to know what’s credible; we can go to a librarian for that. Hakia doesn’t decide whether a resource is credible or not; we use librarians for that”

Tomasz; “If you ask for the capital of Japan we expect a single answer. If you ask about taxes, maybe the IRS is the best source but there are others. If you ask ‘how to get rid of acne’ you expect a lot of results.”

Carla; “We’ve seen three news-making launches in the past month; Wolfram Alpha, Bing, Siri. Is Wolfram the first step towards 2001? How is this engine valuable to those of us who don’t need to solve complex maths?”

Scott; “it’s not the first step… we’ve been working on these problems for a long time. There are a lot of questions people want to ask about the types of data that Wolfram aggregates… We see these things as part of full-search services. Powerset has moved along this path as well, pulling structured data in response to full-text queries.”

William; “Wolfram is a tremendous effort. An interesting example of question answering with structured data. I think people will find uses for it in particular use cases; I spoke to someone who’d used it to calculate when his visa expired, because it could do date calculation. I think there will be use cases in various scenarios; maths, nutrition information, etc… if you remember that it has that sort of information and remember to go to it… However one thing it doesn’t have is a decent back-fill. If it doesn’t have the data, or doesn’t understand the way you asked the query, it gives you nothing. We try to keep quiet and fail over to standard internet search in that sort of circumstance.”

Carla; “Does a semantic search engine know how not to answer a question?”

William; “that’s absolutely fundamental. You need the ability to reliably keep quiet when you don’t have the answer… and fail over reliably to other search services. [True Knowledge does try to do this...] “That requires very high quality semantics.”

Andrew; “One way to characterise the approach of Wolfram Alpha is that it’s a centralised approach. The Wolfram Alpha team goes out to find data and bring it in-house to convert to a standard form. A different approach is to have an ecosystem contributing data in the public eye… It’s not clear yet how much of a value-add is going to come from this centralised knowledge mapping approach. Yahoo! is focussed on the ecosystem approach, and helping people with knowledge to make it available.”

Peter; “Our inclination would be that we don’t want a closed walled garden. We want all the information available to combine in different ways. We want the information to be open, and the tool set to be open for mashing up in different ways.”

Scott; “If Wolfram Alpha hadn’t taken a walled garden approach they might never have launched a product.”

Tomasz; “Wolfram Alpha is great, but it’s not a search engine”

Carla; “Siri… caused a lot of buzz, uses True Knowledge… what are your thoughts?”

Andrew; “To be counter-cultural… the notion of getting much deeper and assisting a user with a task is spot on. We’re going to see much more of that. Search has tended to be stateless. Each query you enter is more or less processed without context. Yahoo! is rolling out more stateful search tools, and other companies will do the same. We expect people to use these tools on lots of devices. Would be expect people to come to the same place for purchase, navigation, etc? Do we expect one interface? There are going to be virtual assistants… I just don’t know if they’re going to be embedded into a search box.”

Scott; “Conversation is the ultimate user interface… but it’s not clear that I want to have a conversation with my laptop during the working day. How do I display the results? But there’s a huge role for conversation and dialogue in refining search and getting a user to their results faster.”

Tomasz; “What is the goal of Siri? If you try to go to broad you become a search engine.”

Scott; “When people have a conversational interface, they won’t speak in keywords.”

Carla; “What are the larger goals for Bing?”

Scott; “Bing is trying to simplify key tasks that people do when they come to a search engine. In travel, health, shopping, we can understand what people are trying to do, and get them to better results faster. The thinking has evolved from ten blue links to the whole page, and organising things to help the user by understanding their tasks.”

Carla; “Peter; what did you think of Bing?”

Peter; “I like the idea of innovation in the user interface. There’s a lot of room for that. There’s been a lot of emphasis on getting the ranking right. You still need to do that, but other things are important too. I’m usually happy with results on my big screen. On a mobile device, I’m usually not happy with the results I get.”

June 16th, 2009

Semantic Technology Conference kicks off with Keynotes from Open Calais and Siri

Posted by Paul Miller @ 9:54 am

Categories: Commercialisation, Open Data, Semantic Web, Semantic Web Companies, Web 2.0, Web 3.0

Tags: Web, Advertisement, Tool, Siri, Tom Tague, Tague, Enterprise Adoption, Virtual Personal Assistant, Virtual Personal Assistant Paradigm, Podcasts

This year’s Semantic Technology Conference got fully underway this morning, with Keynote presentations from Tom Tague of Thomson Reuters’ Open Calais Initiative and Tom Gruber from Siri.

Despite the wider economic situation, attendance for this fifth year of the event feels a little up on last year, and there’s clearly real enthusiasm in the buzzing Halls.

Tague’s Open Calais has been one of the success stories for useful and easy application of semantic technologies beyond a core community of enthusiasts and adopters, and has been covered here and on Cloud of Data a number of times since it launched. Just today, they announced a new set of partners and a postal service that should remove one more perceived barrier for another set of potential adopters.

Speaking to the theme of ‘Web 3.0 - the Web of Me,’ Tague’s abstract suggests;

“The mainstream adoption of Web 2.0 technologies – from RSS feeds to social networks – is hastening the demise of the portal. With each new face on Facebook, and each new Twitter account, our once routine habits and traffic patterns shift. This wave of change in the way we consume, transact and interact on the Web is dis-intermediating ‘destination’ sites of all kinds. Our once centralized content has been atomized.

And yet our fundamental problem persists. We’re overwhelmed with input, yet still can’t find the one thing we need… now.

Semantic technologies – and the content interoperability and Linked Data connections they beget – offer new hope. That is not to say the answer lies in building new search engines, and few would argue for another news aggregator. Rather, our point of inflection lies at the point of consumption. Our task is to simultaneously refine and enrich our digital experience of everything from content and community to commerce.”

Early on, Tague made a ‘non-apologetic statement;’

“People need to start deriving financial benefits from semantic technology. It’s time”

Absolutely!

Tague looks back at the move from ‘Web 1.0,’ described as ‘the last Web we agreed on,’ to ‘Web 2.0,’ which he sees as largely defined by the ‘addition of social.’ Today, he reckons, we are ‘extraordinarily content-rich’, ‘extraordinarily information-poor’ and ‘experientially deficient.’ Despite a wealth of content, we are failing to make the most of it.

‘We’re at the inflection point’ where ‘innovation is exploding’ as we move from developing and inventing toward mainstream adoption of technologies in the semantic technology space. Lots of things will be tried; 90% will fail, but that’s ok.

‘Everyone needs plumbing,’ and that’s what Calais is; semantic plumbing. 13 version releases in 18 months; about 100 presentations, 13,000 registered Open Calais developers, a million great ideas.

Tague reckons the various efforts he comes in contact with fall into six broad buckets;

Tools; Social; Advertising; Search; Publishing; Interface.

First, Enabling Tools. Data Management, Data generation, Databases, Integration and workflow. ‘A big yes.’ ‘We need tools.’ Everyone needs tools, especially as you move from early adopters toward the mainstream. Tools build the bridges that cross the chasm to enterprise adoption.

Enterprise adoption will not happen because it’s cool. Enterprise adoption will not be talked about on Twitter. Enterprise adoption will happen because it’s cheaper/faster/better than what they have just now.

‘Tool vendors need to simplify their story; it’s not about more functionality.’ ‘If I can’t understand your story, then Enterprise IT certainly can’t’

Second, ‘let’s put some frosting on top of social.’ ‘Wouldn’t it be cool if we could…’ Some of it might be cool, but there’s a challenge in monetising social. Adding frosting to the top of an industry that hasn’t worked out its own monetisation is fraught with risk.

‘I haven’t seen a compelling story yet.’

Next, advertising. Almost a dirty word in the semantic technology domain last year. But advertising is fuel, and semantic technologies have a clear role to play in enhancing advertising (see my podcast with Scott Brinker from last year…).

Semantic search; ‘the semantic industry’s brilliant yet under-achieving child.’ The answer to a question no one is asking? General, consumer-facing semantic search… directly competing with Google et al? Not viable.

But vertical search in specific domains… a huge growth opportunity, and people are willing to invest the time, effort and money to make it happen. Room for a handful of players in each domain?

Search; ‘a bifurcated marketplace.’

Publishing; content producers, editorial/aggregation, ‘robotic publishing.’

‘Classic publishers can get enormous value from this technology… not all of the value is in the user experience.’ Much of the value is being found in the back office, making existing data and investments work harder.

Little value in ‘robotic publishing,’ because the content isn’t that readable. Aggregation services like Huffington Post and Daily Me present ‘enormous opportunities.’

Interface; gaming a huge and growing market. $57bn industry. A ’seamless, interactive and responsive experience,’ it’s ‘graphically engaging and fun.’

Zemanta, AdaptiveBlue, Feedly, Apture et al ‘trying to make the consumption experience different’ [better?]. Not suggesting that these are like a game, but many of the drivers may be similar?

“People are on their mobile devices and in the browser; go where the people are.” Which links well to the next keynote…

“Do you care about semantics or about user value?”

“Don’t fund/buy semantic infrastructure beyond what you need; use infrastructure built by others where possible.”

“Think very hard about the user experience; make it compelling and exciting.”

Following Tague’s presentation, Tom Gruber took to the stage to talk about Siri; a company building a Virtual Personal Assistant (with an interesting iPhone app to start things off) that we discussed during a podcast last week. As Gruber’s says;

“We are beginning to see a new interaction paradigm for the web: the Virtual Personal Assistant (VPA). A VPA is task focused: it helps you get things done. You interact with it in natural language, in a conversation. It gets to know you, acts on your behalf, and gets better with time. The VPA paradigm builds on the information and services of the web, with new technical challenges of semantic intent understanding, context awareness, service delegation, and mass personalization.

Siri is a virtual personal assistant for the mobile Internet. Although just in its infancy, Siri can help with some common tasks that human assistants do, such as booking a restaurant, getting tickets to a show, and inviting a friend. We will describe the technology underlying Siri and how it fits in the larger ecosystem of services and data providers. And we will offer a vision of where assistants like Siri are going.”

Tom starts off by showing the Knowledge Navigator video from Apple… which dates all the way back to 1987. Many of the ideas are now coming to fruition; touch screens, a global network, awareness of temporal and social context, speech in and out, a ‘conversational interface,’ ‘delegation of work’ to the machine, and trusted use of personal data.

Is the Knowledge Navigator possible today? ‘No, but we’re getting there.’

Siri is pretty close… in certain well understood contexts, as Gruber shows in a video demo of the evolving iPhone application.

What is a Virtual Personal Assistant? It does things for you; it’s task-oriented. It understands your intent via a conversational metaphor. It gets to know you; it’s not the same for everybody, unlike a search engine.

‘Service delegation [like Siri]; the mother of all mashups’

‘Context is king’ in communicating with a VPA; where am I, what time is it, who am I, etc.

“This really is the beginning of the age of the start of Virtual Assistants.”

Need to solve authorisation/ authentication. If we reach a ‘data commons’ there will be more, better, information to drive choices and decisions.

Tom Tague is a regular member of the Semantic Web Gang podcast, which I moderate. Tom Gruber was the latest guest in my Executive Briefing podcast series.

May 29th, 2009

Bing is not alone; similar techniques alive and well in existing vertical search

Posted by Paul Miller @ 4:10 am

Categories: Commercialisation, Podcasts, Semantic Web Companies, Talking Semantics

Tags: Endeca Technologies Inc., Technique, Robin, Podcasts, Productivity, Search, Internet, Paul Miller

Microsoft’s Bing is attracting plenty of interest today, and perhaps deservedly so as it brings some interesting fresh ideas to the world of generic search engines. Whether it is sufficiently compelling to break our deeply ingrained association of ’search’ with ‘Google’ remains to be seen.

It should be remembered, of course, that broadly similar approaches are already taken to managing and navigating data inside the data centres of large corporations where Autonomy, FAST, Endeca and their peers provide powerful capabilities.

I recorded a podcast with Endeca Chief Scientist Daniel Tunkelang in January and, by chance, spoke with Robin Johnson yesterday. Robin is CEO of FT Search, part of the Financial Times Group, and responsible for a new vertical search tool called Newssift. Newssift combines components from various technology companies (including Endeca, Nstein, Lexalytics and ReelTwo) to offer a useful means of learning more about businesses and the external factors affecting them.

April 15th, 2009

Leigh Dodds talks about Talis Connected Commons

Posted by Paul Miller @ 11:05 am

Categories: Open Data, Podcasts, Semantic Web, Semantic Web Companies, Talking Semantics

Tags: Connected Corp., Podcasts, Internet, Paul Miller

I wrote about Talis’ Connected Commons last month, and today spent some time talking with the company’s Platform Programme Manager, Leigh Dodds.

The conversation has just been released as a podcast which looks at the rationale behind the company’s offer and the specific licensing choices that beneficiaries are asked to make.

Have a listen, and see if the Connected Commons might help your next project.

Disclaimer: Talis is my former employer

April 13th, 2009

True Knowledge API lies at the heart of real business model

Posted by Paul Miller @ 10:01 pm

Categories: Commercialisation, Semantic Web, Semantic Web Companies

Tags: Knowledge, Knowledge Base, True Knowledge, Query API, True Knowledge Home Page, Strategy, Management, Paul Miller

Semantically powered question answering start-up True Knowledge today made its Semantic Search API available for public consumption, taking the next step on the company’s journey out of beta and providing a clear steer as to the way in which they intend to generate revenue.

As the company’s press release notes,

“True Knowledge offers two distinct API services for developers: the ‘Direct Answer API’ and the ‘Query API.’ The Direct Answer API allows developers to leverage True Knowledge’s natural language question answering technology, giving any search site or application the ability to provide a single direct answer for questions asked on any subject in plain English. This is especially well suited to mobile applications where providing a lengthy list of search results may be impractical.

The Query API allows developers to bypass True Knowledge’s natural language translation system and directly query True Knowledge’s knowledge base using a simple query language. This allows automated systems such as web and mobile applications to tap into True Knowledge’s vast machine-understandable knowledge of the world, making them behave more intelligently.”

The company was founded in August 2005 and is based in the British city of Cambridge, at the heart of ‘Silicon Fen.’ A $4m Series A investment round was closed in July 2008, led by Octopus Ventures.

I spoke with CEO William Tunstall-Pedoe ahead of today’s announcement to see how the core knowledge base continues to improve, and to discuss the company’s plans.

For those who haven’t tried it, True Knowledge offers an interesting slant on attempting to answer your question rather than simply return hundreds or thousands of documents that might contain the answer as traditional search engines tend to. Tunstall-Pedoe quoted Google’s Larry Page during our conversation, noting that Page has asserted that

“the perfect search engine would understand exactly what you mean and give back exactly what you want.”

It is this that True Knowledge attempts with their ‘Internet Answer Engine,’ and core to their solution are a comprehensive (137 million facts, and growing) knowledge base, a proprietary system for understanding a query and a powerful inference capability that enables the system to answer questions more reliably. Part of that reliability, as Tunstall-Pedoe frequently stresses, lies in the system’s ability to know when it doesn’t know the answer. Along with a success rate of less than 50% for providing answers to questions, this may seem little more than an academic curiosity, but an ability to reliably know when to fall back to less structured approaches (such as passing the query to Google) is far better than ‘guessing’ or delivering wholly inappropriate responses… especially once the Answer Engine’s capabilities are embedded in some third party site.

True Knowledge’s process of inference also allows the system to cope with ambiguity, and even with contradictory ‘facts.’ During our conversation, we told the system that President Obama was born in Cambridge. It allowed us to make this assertion, but subsequent analysis of the overwhelmingly contradictory data drawn from elsewhere in the knowledge base means that it was deemed to be untrue and flagged as such.

A different query, in which I ask ‘How far is San Jose from SFO?,’ shows both how the system copes with ambiguity and the manner in which supporting facts are drawn from sites such as Metaweb’s Freebase.

The current True Knowledge home page is not going to draw huge numbers of users away from their search engine of choice, but that isn’t really the point. As Tunstall-Pedoe pointed out, the site is intended to showcase the company’s capabilities and facilitate the addition of new knowledge (as well as the millions of facts drawn from Wikipedia, Freebase and a growing body of licensed commercial content, over 120,000 facts have already been added by individuals in the beta programme.) The real utility of True Knowledge will lie in licensing the underlying system for use in vertical and horizontal third party applications, and public availability of the True Knowledge API begins that process. There’s a long way to go in further extending the knowledge base, suggesting that vertical search applications may be the first to sign up; it’s much easier to approach comprehensiveness within a bounded domain than across all areas of knowledge.

The market for semantically enhanced search is growing crowded, and stalwarts of the search industry have been hard at work too, with Google and others getting increasingly good at returning actual answers to factual questions.

Tunstall-Pedoe used a slide to demonstrate the differentiation the company sees between itself and ‘obvious’ competitors such as Wikipedia, Freebase, and hotly anticipated Wolfram Alpha. Key differentiators in the diagram included True Knowledge’s ability to infer (something Wolfram Alpha also claims), its language independence (although currently only available in English, the concept extraction techniques used by True Knowledge should work equally well in other languages), and the system’s reliance upon an internal ontology comprising 20,000 classes (plus biological species, product information, etc). True Knowledge (unsurprisingly) scored far better than the competition, but in a market that also includes the likes of Hakia and Powerset (neither of which could usefully answer my question about San Jose and SFO) the true picture is a lot more complex.

True Knowledge is certainly interesting, and frequently impressive. It remains to be seen whether a Platform proposition will set them firmly on the road to riches, or if they’ll end up finding more success following the same route as Powerset and getting acquired by an existing (enterprise?) search provider.

April 3rd, 2009

AdaptiveBlue updates Glue; I avoid 'sticky' puns with this title

Posted by Paul Miller @ 6:19 am

Categories: Semantic Web Companies, Web 2.0, Web 3.0

Tags: Film, Web Browser, Web Browsers, Wiki, Social Networking, Internet, Online Communications, Marketing, Advertising & Promotion, Paul Miller

New York-based semantic technology startup AdaptiveBlue yesterday unveiled an update to their Glue product, and the world’s technology writers were unable to contain their enthusiasm for the obvious puns. I spoke with AdaptiveBlue’s CEO, Alex Iskold, ahead of the launch to hear about the latest enhancements.

Currently offering an iPhone App and a Firefox browser extension, Glue provides useful functionality in aggregating interactions with identifiable objects such as films and books from across the various sites on which people find with them. As I described when Glue originally launched last year,

“There are plenty of offerings that will put you in touch with your social network on a single site. Glue is interesting because it escapes the tyranny of the site and connects people to things across a growing number of sites. My interactions with Social Networks and the Semantic Web on Amazon.com are visible to members of my network who prefer to shop with Barnes&Noble, and those who are amongst the 32 owners of this book hanging out on LibraryThing. My personal preferences are respected, as I only need to interact with the item on a site of my choosing. Members of my network gain the ‘benefit’ of that interaction without needing to change their habits and visit sites of my choosing. Behind the scenes, semantic technologies are hard at work reconciling the 0387710000 with the 978-0387710006, the 4561465 and the various other ways in which we choose to refer to a single body of intellectual expression. When a match is found, the Glue Firefox plugin does a nice job of subtly highlighting the fact… without getting in the way of whatever task you are trying to complete.”

Since that launch there have been 110,000 downloads of the browser extension, and Alex reports 35,000 ‘active’ users.

Yesterday’s visible additions to the product mostly appear quite superficial, but they’ll be important in converting more of those downloaders to active users, especially once the pool of potential users is expanded by the upcoming version for Internet Explorer.

First, the Glue Bar at the top of a browser window is 25% thinner. Even though the Glue Bar is ‘contextual,’ and only appears on pages where relevant content (a book, a film, a person, etc) is detected, the old version could still intrude quite a long way into the browsing experience. For those who actively engage with the bar only occasionally, it’s now an awful lot less intrusive and therefore more likely to be tolerated in day-to-day browsing for the benefits it brings.

More significantly, the ‘2 cent’ comments that users of the previous version were able to make are now aggregated and made far more visible to other Glue users via what Iskold described as ‘Connected Conversations.’ As with the core Glue offer, users’ 2 cent comments about a film on Netflix, Wikipedia or IMDB are pulled together and made visible to their peers, regardless of the site on which they happen to be viewing details of the same film. These comments can also be shared via a user’s social graph on Twitter, Tumblr and Friendfeed, reaching a community beyond Glue.

By aggregating the interactions of every Glue user with items scattered across the Web, Glue is also now able to compile - and display - lists of the most popular books, films, etc. Unlike traditional measures that might track rentals from Netflix and sales from Amazon, Glue is able to measure a far more complex set of interactions with a given resource. Users might buy from Amazon or rent from Netflix. They might also demonstrate their interest by reading about a film on Wikipedia or IMDB… and Glue would track those interactions too. Below, for example, we can see that Knowing is the most popular film on Glue this week.

These lists are calculated daily, and are based upon the aggregate of all user activity over the preceding seven days.

The list of sites that Glue understands is growing, but remains heavily biased to the US market. Alex suggested that the team were keen to consolidate their US position before devoting more attention to adding international sources. However, with a Glue API to follow their Internet Explorer release, it’s possible that a sufficiently motivated community will soon be able to start adding some of these new resources for themselves…

March 30th, 2009

Growing the Linked Data pool, with the Talis Connected Commons

Posted by Paul Miller @ 12:46 pm

Categories: Open Data, Semantic Web, Semantic Web Companies

Tags: Data, Amazon.com Inc., Semantic Web, Cloud Computing, Web Services, Internet, Enterprise Software, Software, Paul Miller

Back in December of 2008, I wrote about a new initiative from Amazon to make large sets of public data more accessible. Amazon offered to mount the data for free, and for developers writing applications elsewhere in the Amazon Web Services ecosystem even the bandwidth cost of communicating with GenBank, the PubChem Library, the US Census and similar resources was zero. As I wrote at the time,

“By offering free hosting for public data, then, Amazon are doing the wider community a huge service. Much of the data there today is reasonably readily available from other sources, so the biggest immediate benefits are those of speed and cost… For existing or potential users of Amazon’s Web Services to power their applications, this is yet another reason to consider Amazon.”

Over the weekend, Talis made a similar offer to host public domain data (licensed under Creative Commons’ CC0 or Open Data Commons’ PDDL).

What’s interesting about the company’s ‘Connected Commons‘ is that the data sits in their Semantic Web Platform; all of the APIs for querying and managing data are at your disposal, free of charge.

Sir Tim Berners-Lee recently called on the holders of data to loosen their grip, demanding Raw Data, Now! For those prepared to take that step, and unsure where to go next, offers such as those from Talis and Amazon are certainly worth a very close look.

Disclaimer: Talis is my former employer